[jira] [Commented] (JENA-801) When the server is under load, many queries are piling up and seems to be in some kind of dead lock.

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14210093#comment-14210093
 ] 

Simon Helsen commented on JENA-801:
---

Bala, when running in memory mapped I/O, you need to *reduce* the Java heap 
because allocations will happen in the native heap of the app (which is 
controlled by the OS and outside the -Xmx). If there isn't enough native heap, 
you will run into an OOME as well and it may not be immediately apparent from 
the exception that the OOME was caused by native heap exhaustion. 

> When the server is under load, many queries are piling up and seems to be in 
> some kind of dead lock.
> 
>
> Key: JENA-801
> URL: https://issues.apache.org/jira/browse/JENA-801
> Project: Apache Jena
>  Issue Type: Bug
>  Components: TDB
>Affects Versions: TDB 0.9.4, Jena 2.11.2
>Reporter: Bala Kolla
> Attachments: 
> ThreadLocksInBlockMgrJournalAfterGuavaCacheInNodeTable.htm, 
> TracesWithManyItersOfBlockMgrJournal_Valid_Method.txt, 
> TracesWithManyItersOfBlockMgrJournal_getRead_Method.txt, 
> WAITDataReportShowingTheLockContention.zip, 
> WAITDataReportShowingTheLockContentionWithoutQueryFilter.zip
>
>
> We were testing our server with repositories of varied sizes and in almost 
> all the cases when the server peaks its capacity (of maximum number of users 
> it can support), It seems like the queries are piling up because of the lock 
> contention in NodeTableCache.
> Here are some details about the repository..
> size of indices on disk - 150GB
> type of hard disk used - SSD and HDD with high RAM (seeing the same result in 
> both the cases)
> OS - Linux
> Details on the user load;
> We are trying to simulate a very active user load where all the users are 
> executing many usecases that would result in many queries and updates on TDB.
> I would like to know what are the possible solutions to work around and avoid 
> this situation. I am thinking of the following, please let me know if there 
> is any other way to work around this bottleneck.
> Control the updates to the triple store so that we only do it when there are 
> not many queries pending. We would have to experiment how this impact the 
> usecases..
> Is there any other way to make this lock contention go away? Can we have 
> multiple instances of this cache? For example many (90%) of our queries are 
> executed with a query scope (per project). So, can we have a separate 
> NodeTable cache for each query scope (project in our case) and one for 
> global? 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (JENA-801) When the server is under load, many queries are piling up and seems to be in some kind of dead lock.

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14185358#comment-14185358
 ] 

Simon Helsen commented on JENA-801:
---

The project Bala is working in still uses direct mode (even in 64 bit) AFAIK. 
This had to do with the file locking problems caused by Windows and the desire 
to have platform consistency. Bala, it would be worth testing with mapped, even 
if it is only an option under linux

> When the server is under load, many queries are piling up and seems to be in 
> some kind of dead lock.
> 
>
> Key: JENA-801
> URL: https://issues.apache.org/jira/browse/JENA-801
> Project: Apache Jena
>  Issue Type: Bug
>  Components: TDB
>Affects Versions: TDB 0.9.4, Jena 2.11.2
>Reporter: Bala Kolla
> Attachments: 
> ThreadLocksInBlockMgrJournalAfterGuavaCacheInNodeTable.htm, 
> WAITDataReportShowingTheLockContention.zip, 
> WAITDataReportShowingTheLockContentionWithoutQueryFilter.zip
>
>
> We were testing our server with repositories of varied sizes and in almost 
> all the cases when the server peaks its capacity (of maximum number of users 
> it can support), It seems like the queries are piling up because of the lock 
> contention in NodeTableCache.
> Here are some details about the repository..
> size of indices on disk - 150GB
> type of hard disk used - SSD and HDD with high RAM (seeing the same result in 
> both the cases)
> OS - Linux
> Details on the user load;
> We are trying to simulate a very active user load where all the users are 
> executing many usecases that would result in many queries and updates on TDB.
> I would like to know what are the possible solutions to work around and avoid 
> this situation. I am thinking of the following, please let me know if there 
> is any other way to work around this bottleneck.
> Control the updates to the triple store so that we only do it when there are 
> not many queries pending. We would have to experiment how this impact the 
> usecases..
> Is there any other way to make this lock contention go away? Can we have 
> multiple instances of this cache? For example many (90%) of our queries are 
> executed with a query scope (per project). So, can we have a separate 
> NodeTable cache for each query scope (project in our case) and one for 
> global? 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (JENA-804) Jena is not reusing already allocated space on the file system which results in large amounts of disk space reserved by Jena files

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14181790#comment-14181790
 ] 

Simon Helsen commented on JENA-804:
---

There are different teams in IBM using TDB. Keith belongs to a team which uses 
TDB in mapped mode AFAIK. 

> Jena is not reusing already allocated space on the file system which results 
> in large amounts of disk space reserved by Jena files
> --
>
> Key: JENA-804
> URL: https://issues.apache.org/jira/browse/JENA-804
> Project: Apache Jena
>  Issue Type: Bug
>  Components: Jena
>Affects Versions: TDB 1.0.2
> Environment: Windows 7, IBM JRE 1.7, Tomcat 7.0.54
>Reporter: Keith Wells
>
> We have a product based on Jena TDB where we insert quads to Jena TDB along 
> with the deletion of quads.  We understand the performance over space 
> architectural decision to not clean up deleted nodeids from the indexes. But 
> the usage of disk space appears that Jena TDB is not reusing allocated space 
> which had been allocated by Jena previously.  Based on this comment there 
> appears to be something that is not correct on file space utilization, 
> http://mail-archives.apache.org/mod_mbox/jena-users/201310.mbox/%3cce7d7929.2a707%25rve...@dotnetrdf.org%3E:
>  "The indexes won't shrink - TDB never gives disk space back to the OS -  but 
> disk space is reused when reallocated within the same JVM.".
> In this scenario on the same JVM with NO server stops or starts, we add 27765 
> graphs to IndexTdb and immediately remove them,  repeating this process 
> several times. 
>  MB   Bytes   Diff (Bytes)
> Start   193   203239424   
>   
> Reindex 5 249 262066176   58826752
> Reindex 6 249 262086656   20480
> Reindex 10298 312500224   50413568
> Reindex 11298 312520704   20480
> Reindex 12298 312541184   20480
> Reindex 13298 312586240   45056
> Reindex 14306 320995328   8409088
> Reindex 15330 346181632   25186304
> Reindex 16330 346198538   16906
> Reindex 17346 362999808   16801270
> Reindex 18346 363020288   20480
> Reindex 19346 363040768   20480
> Reindex 20346 363061248   20480
> Reindex 21346 363081728   20480
> Reindex 22354 371490816   8409088
> Reindex 23378 396677120   25186304
>   
> End   193 203239424   
> The system starts with 193MB of data allocated by indexTdb.  A reindex 
> consists of a remove followed by an add of these graphs. As you can see from 
> the data there is a dramatic increase in the size of indexTdb on the disk 
> after repeadedly removing and adding graphs.  After Reindex 23, there is 378 
> MB of disk space used.  If Jena TDB reused allocated space there would be no 
> need to allocate more space other than what is used by deleted node ids 
> (unless nodeid storage is eating all of this space?).  Jena does not appear 
> to be reusing the allocated disk space.  At the very end of this scenario, we 
> exported the nquads and reloaded them to show the original disk space was 
> 193MB back to where it started. 
> We believe Jena TDB is not reusing the space allocated by the TDB file system 
> within the same JVM.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (JENA-775) Jena TDB doesn't seem to release all file handles when dataset is closed

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14120087#comment-14120087
 ] 

Simon Helsen commented on JENA-775:
---

Matt, just for the record, the explicit gc call did not always work, even 
though it appeared to work most of the time. But then again, it could be that 
an IBM JRE is still acting slightly different. 

> Jena TDB doesn't seem to release all file handles when dataset is closed
> 
>
> Key: JENA-775
> URL: https://issues.apache.org/jira/browse/JENA-775
> Project: Apache Jena
>  Issue Type: Bug
>  Components: TDB
>Affects Versions: TDB 1.0.2
> Environment: Windows
>Reporter: Matthew Jarvis
>
> Jena TDB doesn't allow renaming of a TDB directory after a dataset has been 
> opened and a model has been added.  It appears as though there is an open 
> file handle that is preventing the rename from succeeding.  The following 
> JUnit test fails:
> {noformat}
> import java.io.File;
> import junit.framework.Assert;
> import org.junit.Test;
> import com.hp.hpl.jena.query.Dataset;
> import com.hp.hpl.jena.rdf.model.Model;
> import com.hp.hpl.jena.tdb.StoreConnection;
> import com.hp.hpl.jena.tdb.TDBFactory;
> import com.hp.hpl.jena.tdb.base.file.Location;
> import com.hp.hpl.jena.vocabulary.RDFS;
> @Test
> public void testRename() {
>   String path = "C:\\tmp\\indexTdb";
>   Dataset ds = TDBFactory.createDataset(path);
>   StoreConnection.make(path);
>   Model model = ds.getNamedModel(RDFS.Class.getURI());
>   ds.addNamedModel(RDFS.Class.getURI(), model);
>   ds.close();
>   StoreConnection.release(new Location(path));
>   File tdbRename = new File(path);
>   Assert.assertTrue(tdbRename.renameTo(new 
> File("C:\\tmp\indexTdbMoved")));
> }
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (JENA-597) IRIResolverNormal needs thread safe CacheLRU

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13832944#comment-13832944
 ] 

Simon Helsen commented on JENA-597:
---

this stack trace matches 2.11. There are other calls to  

public IRI resolveSilent(String relURI):

In ParserProfileChecker: public IRI makeIRI(String uriStr, long line, long col)
In IRIResolverNormal:  public IRI resolve(String relURI) 
In IRIResolver: public String resolveToStringSilent(String uriStr) 

not sure which one could be involved in this situation



> IRIResolverNormal needs thread safe CacheLRU
> 
>
> Key: JENA-597
> URL: https://issues.apache.org/jira/browse/JENA-597
> Project: Apache Jena
>  Issue Type: Bug
>  Components: ARQ
>Affects Versions: Jena 2.11.0
> Environment: IBM Websphere 8 JRE
>Reporter: Scott Patterson
> Attachments: IRIResolverTest.java
>
>
> The following exception may occur when more than one thread requires access 
> to the org.apache.jena.atlas.lib.cache.CacheLRU embedded in IRIResolverNormal:
> Caused by: java.lang.NullPointerException
>   at java.util.LinkedHashMap.get(LinkedHashMap.java:339)
>   at org.apache.jena.atlas.lib.cache.CacheLRU.get(CacheLRU.java:53)
>   at 
> org.apache.jena.riot.system.IRIResolver$IRIResolverNormal.resolveSilent(IRIResolver.java:427)
>   at 
> org.apache.jena.riot.system.IRIResolver$IRIResolverNormal.(IRIResolver.java:383)
>   at org.apache.jena.riot.system.IRIResolver.create(IRIResolver.java:210)
>   at org.apache.jena.riot.system.RiotLib.profile(RiotLib.java:141)
>   at org.apache.jena.riot.system.RiotLib.profile(RiotLib.java:130)
>   at org.apache.jena.riot.lang.LangRDFXML.(LangRDFXML.java:104)
>   at org.apache.jena.riot.lang.LangRDFXML.create(LangRDFXML.java:74)
>   at org.apache.jena.riot.RiotReader.createParser(RiotReader.java:128)
>   at 
> org.apache.jena.riot.RDFParserRegistry$ReaderRIOTFactoryImpl$1.read(RDFParserRegistry.java:141)
>   at org.apache.jena.riot.RDFDataMgr.process(RDFDataMgr.java:818)
>   at org.apache.jena.riot.RDFDataMgr.read(RDFDataMgr.java:258)
>   at org.apache.jena.riot.RDFDataMgr.read(RDFDataMgr.java:244)
>   at 
> org.apache.jena.riot.adapters.RDFReaderRIOT.read(RDFReaderRIOT.java:69)
>   at com.ibm.team.jis.lqe.resource.RDFEntity.getModel(RDFEntity.java:361)
>   ... 39 more
> This may be related to the problem reported that is suppose to be fixed by 
> the cloned issue. It looks to be the same stack trace. I've attached a test 
> to reproduce. Works with Oracle Oracle JRE 1.6.0.27 but not IBM jre. Keep 
> stopping and restarting the test until the NPE happens. NPE usually happens 
> right off the start.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (JENA-546) Query timeout not observed in OPTIONAL eval

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13778044#comment-13778044
 ] 

Simon Helsen commented on JENA-546:
---

Andy, just for the record. The issue is not whether the query is efficient or 
well-written. The problem we face is that one badly written query can bring 
down an entire server. This happens when it keeps running for a very long time 
and commandeers all CPUs in the process. 

> Query timeout not observed in OPTIONAL eval
> ---
>
> Key: JENA-546
> URL: https://issues.apache.org/jira/browse/JENA-546
> Project: Apache Jena
>  Issue Type: Bug
>  Components: TDB
>Affects Versions: TDB 0.10.1, TDB 1.0.0
>Reporter: Mark Buquor
> Attachments: query1.txt
>
>
> We've found two queries so far that fail to time out in a reasonable time 
> after the timeout elapses. The severity of the issue varies. In one case, the 
> query completes without error ~30 minutes after it should have aborted. In 
> the other case, the query runs indefinitely.
> Beyond wasted resources, the critical issue for us is that queries must 
> reliably abort in order for TDB to flush the writeback queue. Failure to 
> flush the queue leads to stack overflows and heap exhaustion.
> I haven't been able to isolate a testcase yet, but in both cases, the problem 
> appears to be isolated to an OPTIONAL block. Running under the debugger, the 
> process was suspended after the query should have aborted. I placed comments 
> in the stack below to show 1) the last iterator in the stack with abortFlag 
> == true and 2) the QueryIterPeek directly above it has abortIterator == 
> false. All iterators above the first comment have not been aborted, so it 
> appears that the abort signal does not propagate sufficiently.
> I'm not sure if this affects ARQ in general. I'm filing this against TDB 
> because the QueryIterTDB below the comments is the last to have 
> requestCancel() called. Its killList includes the two IterAbortables directly 
> above it. Nothing else above that QueryIterTDB is aborted.
> {noformat}
> Thread [main] (Suspended) 
>   
> com.hp.hpl.jena.tdb.index.bplustree.BPTreeNode.recordsPageId(com.hp.hpl.jena.tdb.index.bplustree.BPTreeNode,
>  com.hp.hpl.jena.tdb.base.record.Record) line: 277  
>   
> com.hp.hpl.jena.tdb.index.bplustree.BPlusTree.iterator(com.hp.hpl.jena.tdb.index.bplustree.BPTreeNode,
>  com.hp.hpl.jena.tdb.base.record.Record, 
> com.hp.hpl.jena.tdb.base.record.Record) line: 378
>   
> com.hp.hpl.jena.tdb.index.bplustree.BPlusTree.iterator(com.hp.hpl.jena.tdb.base.record.Record,
>  com.hp.hpl.jena.tdb.base.record.Record) line: 366
>   
> com.hp.hpl.jena.tdb.index.TupleIndexRecord.findWorker(org.apache.jena.atlas.lib.Tuple,
>  boolean, boolean) line: 164
>   
> com.hp.hpl.jena.tdb.index.TupleIndexRecord.findOrScan(org.apache.jena.atlas.lib.Tuple)
>  line: 84   
>   
> com.hp.hpl.jena.tdb.index.TupleIndexRecord.performFind(org.apache.jena.atlas.lib.Tuple)
>  line: 78  
>   
> com.hp.hpl.jena.tdb.index.TupleIndexRecord(com.hp.hpl.jena.tdb.index.TupleIndexBase).find(org.apache.jena.atlas.lib.Tuple)
>  line: 91   
>   
> com.hp.hpl.jena.tdb.index.TupleTable.find(org.apache.jena.atlas.lib.Tuple)
>  line: 197  
>   
> com.hp.hpl.jena.tdb.nodetable.NodeTupleTableConcrete.find(org.apache.jena.atlas.lib.Tuple)
>  line: 169  
>   
> com.hp.hpl.jena.tdb.solver.StageMatchTuple.makeNextStage(com.hp.hpl.jena.tdb.solver.BindingNodeId)
>  line: 91 
>   
> com.hp.hpl.jena.tdb.solver.StageMatchTuple.makeNextStage(java.lang.Object) 
> line: 37 
>   
> com.hp.hpl.jena.tdb.solver.StageMatchTuple(org.apache.jena.atlas.iterator.RepeatApplyIterator).hasNext()
>  line: 49
>   com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext() line: 
> 193   
>   org.apache.jena.atlas.iterator.Iter$4.hasNext() line: 293   
>   
> com.hp.hpl.jena.tdb.solver.QueryIterTDB(com.hp.hpl.jena.sparql.engine.iterator.QueryIterPlainWrapper).hasNextBinding()
>  line: 54 
>   
> com.hp.hpl.jena.tdb.solver.QueryIterTDB(com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase).hasNext()
>  line: 112   
>   
> com.hp.hpl.jena.sparql.engine.iterator.QueryIterDefaulting.hasNextBinding() 
> line: 54
>   
> com.hp.hpl.jena.sparql.engine.iterator.QueryIterDefaulting(com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase).hasNext()
>  line: 112
>   
> com.hp.hpl.jena.sparql.engine.main.iterator.QueryIterOptionalIndex(com.hp.hpl.jena.sparql.engine.iterator.QueryIterRepeatApply).hasNextBinding()
>  line: 81   
>   
> com.hp.hpl.jena.sparql.engine.main.iterator.QueryIterOptionalIndex(com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase).hasNext()
>  line: 1

[jira] [Commented] (JENA-467) IRIResolverNormal should use synchronized CacheLRU

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13676389#comment-13676389
 ] 

Simon Helsen commented on JENA-467:
---

looks like the alternative patches (i.e. iriresolver-patch.txt and 
iriresolver-patch-2.txt) instead of my original proposal works ok, at least, my 
test is not hitting the NPE anymore.

> IRIResolverNormal should use synchronized CacheLRU
> --
>
> Key: JENA-467
> URL: https://issues.apache.org/jira/browse/JENA-467
> Project: Apache Jena
>  Issue Type: Bug
>  Components: ARQ
>Affects Versions: Jena 2.10.1
> Environment: Any IBM JRE 6 or higher
>Reporter: Simon Helsen
>Priority: Critical
> Attachments: iriresolver-patch-2.txt, iriresolver-patch.txt, patch.txt
>
>
> The following exception may occur on an IBM JRE 6 or higher when more than 
> one thread requires access to the org.apache.jena.atlas.lib.cache.CacheLRU 
> embedded in IRIResolverNormal:
> Caused by: java.lang.NullPointerException
>   at java.util.LinkedHashMap.get(LinkedHashMap.java:337)
>   at org.apache.jena.atlas.lib.cache.CacheLRU.get(CacheLRU.java:53)
>   at 
> org.apache.jena.riot.system.IRIResolver$IRIResolverNormal.resolveSilent(IRIResolver.java:403)
>   at 
> org.apache.jena.riot.system.IRIResolver$IRIResolverNormal.(IRIResolver.java:359)
>   at org.apache.jena.riot.system.IRIResolver.create(IRIResolver.java:212)
>   at org.apache.jena.riot.system.RiotLib.profile(RiotLib.java:141)
>   at org.apache.jena.riot.system.RiotLib.profile(RiotLib.java:130)
>   at org.apache.jena.riot.system.RiotLib.profile(RiotLib.java:117)
>   at 
> org.apache.jena.riot.RiotReader.createParserTurtle(RiotReader.java:310)
>   at org.apache.jena.riot.RiotReader.createParser(RiotReader.java:142)
>   at org.apache.jena.riot.RiotReader.createParser(RiotReader.java:133)
>   at 
> org.apache.jena.riot.RDFParserRegistry$ReaderRIOTFactoryImpl$1.read(RDFParserRegistry.java:141)
>   at org.apache.jena.riot.RDFDataMgr.process(RDFDataMgr.java:760)
>   at org.apache.jena.riot.RDFDataMgr.read(RDFDataMgr.java:258)
>   at org.apache.jena.riot.RDFDataMgr.read(RDFDataMgr.java:244)
>   ... 20 more
> The problem seems to only occurs on IBM JRE 6 or higher as its implementation 
> of LinkedHashMap is not thread safe. Note that the JDK specification does not 
> required LinkedHashMap to be thread safe. 
> The solution is to make use of CacheFactory.createSync when creating the 
> Cache in the IRIResolverNormal. Patch attached

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-467) IRIResolverNormal should use synchronized CacheLRU

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13676372#comment-13676372
 ] 

Simon Helsen commented on JENA-467:
---

I have a local test which _somewhat_ reliably reproduces the problem. 
Unfortunately, right now, it is not in a sharable form. I presume a more 
minimal test needs to spawn a number of threads which read/writes models and 
require some form of base uri resolution. But I can apply your patches and test 
them with what I have. I actually anticipated my proposal was a bit 
heavy-handed.

> IRIResolverNormal should use synchronized CacheLRU
> --
>
> Key: JENA-467
> URL: https://issues.apache.org/jira/browse/JENA-467
> Project: Apache Jena
>  Issue Type: Bug
>  Components: ARQ
>Affects Versions: Jena 2.10.1
> Environment: Any IBM JRE 6 or higher
>Reporter: Simon Helsen
>Priority: Critical
> Attachments: iriresolver-patch-2.txt, iriresolver-patch.txt, patch.txt
>
>
> The following exception may occur on an IBM JRE 6 or higher when more than 
> one thread requires access to the org.apache.jena.atlas.lib.cache.CacheLRU 
> embedded in IRIResolverNormal:
> Caused by: java.lang.NullPointerException
>   at java.util.LinkedHashMap.get(LinkedHashMap.java:337)
>   at org.apache.jena.atlas.lib.cache.CacheLRU.get(CacheLRU.java:53)
>   at 
> org.apache.jena.riot.system.IRIResolver$IRIResolverNormal.resolveSilent(IRIResolver.java:403)
>   at 
> org.apache.jena.riot.system.IRIResolver$IRIResolverNormal.(IRIResolver.java:359)
>   at org.apache.jena.riot.system.IRIResolver.create(IRIResolver.java:212)
>   at org.apache.jena.riot.system.RiotLib.profile(RiotLib.java:141)
>   at org.apache.jena.riot.system.RiotLib.profile(RiotLib.java:130)
>   at org.apache.jena.riot.system.RiotLib.profile(RiotLib.java:117)
>   at 
> org.apache.jena.riot.RiotReader.createParserTurtle(RiotReader.java:310)
>   at org.apache.jena.riot.RiotReader.createParser(RiotReader.java:142)
>   at org.apache.jena.riot.RiotReader.createParser(RiotReader.java:133)
>   at 
> org.apache.jena.riot.RDFParserRegistry$ReaderRIOTFactoryImpl$1.read(RDFParserRegistry.java:141)
>   at org.apache.jena.riot.RDFDataMgr.process(RDFDataMgr.java:760)
>   at org.apache.jena.riot.RDFDataMgr.read(RDFDataMgr.java:258)
>   at org.apache.jena.riot.RDFDataMgr.read(RDFDataMgr.java:244)
>   ... 20 more
> The problem seems to only occurs on IBM JRE 6 or higher as its implementation 
> of LinkedHashMap is not thread safe. Note that the JDK specification does not 
> required LinkedHashMap to be thread safe. 
> The solution is to make use of CacheFactory.createSync when creating the 
> Cache in the IRIResolverNormal. Patch attached

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (JENA-467) IRIResolverNormal should use synchronized CacheLRU

[Simon Helsen (JIRA)]

Simon Helsen created JENA-467:
-

 Summary: IRIResolverNormal should use synchronized CacheLRU
 Key: JENA-467
 URL: https://issues.apache.org/jira/browse/JENA-467
 Project: Apache Jena
  Issue Type: Bug
  Components: ARQ
Affects Versions: Jena 2.10.1
 Environment: Any IBM JRE 6 or higher
Reporter: Simon Helsen
Priority: Critical
 Attachments: patch.txt

The following exception may occur on an IBM JRE 6 or higher when more than one 
thread requires access to the org.apache.jena.atlas.lib.cache.CacheLRU embedded 
in IRIResolverNormal:

Caused by: java.lang.NullPointerException
at java.util.LinkedHashMap.get(LinkedHashMap.java:337)
at org.apache.jena.atlas.lib.cache.CacheLRU.get(CacheLRU.java:53)
at 
org.apache.jena.riot.system.IRIResolver$IRIResolverNormal.resolveSilent(IRIResolver.java:403)
at 
org.apache.jena.riot.system.IRIResolver$IRIResolverNormal.(IRIResolver.java:359)
at org.apache.jena.riot.system.IRIResolver.create(IRIResolver.java:212)
at org.apache.jena.riot.system.RiotLib.profile(RiotLib.java:141)
at org.apache.jena.riot.system.RiotLib.profile(RiotLib.java:130)
at org.apache.jena.riot.system.RiotLib.profile(RiotLib.java:117)
at 
org.apache.jena.riot.RiotReader.createParserTurtle(RiotReader.java:310)
at org.apache.jena.riot.RiotReader.createParser(RiotReader.java:142)
at org.apache.jena.riot.RiotReader.createParser(RiotReader.java:133)
at 
org.apache.jena.riot.RDFParserRegistry$ReaderRIOTFactoryImpl$1.read(RDFParserRegistry.java:141)
at org.apache.jena.riot.RDFDataMgr.process(RDFDataMgr.java:760)
at org.apache.jena.riot.RDFDataMgr.read(RDFDataMgr.java:258)
at org.apache.jena.riot.RDFDataMgr.read(RDFDataMgr.java:244)
... 20 more

The problem seems to only occurs on IBM JRE 6 or higher as its implementation 
of LinkedHashMap is not thread safe. Note that the JDK specification does not 
required LinkedHashMap to be thread safe. 

The solution is to make use of CacheFactory.createSync when creating the Cache 
in the IRIResolverNormal. Patch attached

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (JENA-467) IRIResolverNormal should use synchronized CacheLRU

[Simon Helsen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/JENA-467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Simon Helsen updated JENA-467:
--

Attachment: patch.txt

> IRIResolverNormal should use synchronized CacheLRU
> --
>
> Key: JENA-467
> URL: https://issues.apache.org/jira/browse/JENA-467
> Project: Apache Jena
>  Issue Type: Bug
>  Components: ARQ
>Affects Versions: Jena 2.10.1
> Environment: Any IBM JRE 6 or higher
>Reporter: Simon Helsen
>Priority: Critical
> Attachments: patch.txt
>
>
> The following exception may occur on an IBM JRE 6 or higher when more than 
> one thread requires access to the org.apache.jena.atlas.lib.cache.CacheLRU 
> embedded in IRIResolverNormal:
> Caused by: java.lang.NullPointerException
>   at java.util.LinkedHashMap.get(LinkedHashMap.java:337)
>   at org.apache.jena.atlas.lib.cache.CacheLRU.get(CacheLRU.java:53)
>   at 
> org.apache.jena.riot.system.IRIResolver$IRIResolverNormal.resolveSilent(IRIResolver.java:403)
>   at 
> org.apache.jena.riot.system.IRIResolver$IRIResolverNormal.(IRIResolver.java:359)
>   at org.apache.jena.riot.system.IRIResolver.create(IRIResolver.java:212)
>   at org.apache.jena.riot.system.RiotLib.profile(RiotLib.java:141)
>   at org.apache.jena.riot.system.RiotLib.profile(RiotLib.java:130)
>   at org.apache.jena.riot.system.RiotLib.profile(RiotLib.java:117)
>   at 
> org.apache.jena.riot.RiotReader.createParserTurtle(RiotReader.java:310)
>   at org.apache.jena.riot.RiotReader.createParser(RiotReader.java:142)
>   at org.apache.jena.riot.RiotReader.createParser(RiotReader.java:133)
>   at 
> org.apache.jena.riot.RDFParserRegistry$ReaderRIOTFactoryImpl$1.read(RDFParserRegistry.java:141)
>   at org.apache.jena.riot.RDFDataMgr.process(RDFDataMgr.java:760)
>   at org.apache.jena.riot.RDFDataMgr.read(RDFDataMgr.java:258)
>   at org.apache.jena.riot.RDFDataMgr.read(RDFDataMgr.java:244)
>   ... 20 more
> The problem seems to only occurs on IBM JRE 6 or higher as its implementation 
> of LinkedHashMap is not thread safe. Note that the JDK specification does not 
> required LinkedHashMap to be thread safe. 
> The solution is to make use of CacheFactory.createSync when creating the 
> Cache in the IRIResolverNormal. Patch attached

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-440) Query timeout does not always correctly move to the second query timeout after first row yielded

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13636467#comment-13636467
 ] 

Simon Helsen commented on JENA-440:
---

is (N,Infinity) meaningful? It essentially means you're only timing out if it 
takes too long to get the first result. Why does that matter to a client? I 
just don't see the use case

> Query timeout does not always correctly move to the second query timeout 
> after first row yielded
> 
>
> Key: JENA-440
> URL: https://issues.apache.org/jira/browse/JENA-440
> Project: Apache Jena
>  Issue Type: Bug
>  Components: ARQ
>Affects Versions: Jena 2.10.0
>Reporter: Andy Seaborne
>Assignee: Andy Seaborne
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-440) Query timeout does not always correctly move to the second query timeout after first row yielded

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13636452#comment-13636452
 ] 

Simon Helsen commented on JENA-440:
---

I have to agree with Andy on this. We do not use the double timeout value and 
we have implemented our own query monitor which manages our timeout semantics, 
but IMO when you specify only one value, you expect the query to be completely 
finished by then, so (N,0), not (N,infinity).

> Query timeout does not always correctly move to the second query timeout 
> after first row yielded
> 
>
> Key: JENA-440
> URL: https://issues.apache.org/jira/browse/JENA-440
> Project: Apache Jena
>  Issue Type: Bug
>  Components: ARQ
>Affects Versions: Jena 2.10.0
>Reporter: Andy Seaborne
>Assignee: Andy Seaborne
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-408) SDB should not attempt to execute multiple update statements in one jdbc statement

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13632875#comment-13632875
 ] 

Simon Helsen commented on JENA-408:
---

yes, perhaps it was a bit too quick. The alternative is to avoid these combined 
statements but that requires more work as you need some datastructure to pass 
around multiple SQL statements.Note: we investigated the use of SDB for a 
special use-case, but abandoned this approach after all because SDB is just too 
slow especially for updates. So for us, this defect has become less important.

> SDB should not attempt to execute multiple update statements in one jdbc 
> statement
> --
>
> Key: JENA-408
> URL: https://issues.apache.org/jira/browse/JENA-408
> Project: Apache Jena
>  Issue Type: Bug
>  Components: SDB
>Affects Versions: SDB 1.3.6
>Reporter: Simon Helsen
> Fix For: SDB 1.3.6
>
> Attachments: patch.txt
>
>
> Some of the db providers (I noticed in db2, but there may be others) attempt 
> to execute update statements by separating them with a semi-colon. This is 
> not supported by the jdbc spec and most jdbc drivers won't be able to handle 
> this correctly. 
> I am not entirely sure why such statements are present in SDB but I suspect 
> that old jdbc 3 drivers for db2 permitted this. The fix is relatively simple 
> and a patch is being attached

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-432) Filter optimization messes up when a nested select is present in the same block

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13630356#comment-13630356
 ] 

Simon Helsen commented on JENA-432:
---

I'm not sure what I know more than what I said. The following query:

SELECT ?test ?p1
WHERE {
  ?test ?p1 ?s2.
  FILTER ( ?test =  || ?test =  )
  { SELECT ?s2
{ ?s2 ?p2 ?o2 }
  }
} 

produces 

(project (?test ?s1)
  (disjunction
(table empty)
(table empty))) 

instead, I think it should produce 

(project (?test ?p1)
  (disjunction
(assign ((?test ))
(join
(bgp (triple  ?p1 ?s2))
(project (?s2)
(bgp (triple ?/s2 ?/p2 ?/o2)
(assign ((?test ))
(join
(bgp (triple  ?p1 ?s2))
(project (?s2)
(bgp (triple ?/s2 ?/p2 ?/o2)
  ))

(that would be the test case)

> Filter optimization messes up when a nested select is present in the same 
> block
> ---
>
> Key: JENA-432
> URL: https://issues.apache.org/jira/browse/JENA-432
> Project: Apache Jena
>  Issue Type: Bug
>  Components: ARQ
>Affects Versions: Jena 2.10.0
>Reporter: Simon Helsen
>
> The following test query:
> SELECT ?test ?s1
> WHERE {
>   ?test ?p1 ?o1.
>   FILTER ( ?test =  || ?test =  )  
>   OPTIONAL {
> SELECT ?s1
> { ?s1 ?p2 ?o2 }
>   }
> }
> produces the following plan:
> (project (?test ?s1)
>   (disjunction
> (table empty)
> (table empty)))
> Something goes wrong with the FILTER expansion. As a workaround, we observed 
> that the following variation:
> SELECT ?test ?s1
> WHERE {
>   { ?test ?p1 ?o1.
> FILTER ( ?test =  || ?test =  ) 
>  
>   }
>   OPTIONAL {
> SELECT ?s1
> { ?s1 ?p2 ?o2 }
>   }
> }
> produces the correct plan:
> (project (?test ?s1)
>   (leftjoin
> (disjunction
>   (assign ((?test ))
> (bgp (triple  ?p1 ?o1)))
>   (assign ((?test ))
> (bgp (triple  ?p1 ?o1
> (project (?s1)
>   (bgp (triple ?s1 ?/p2 ?/o2)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-432) Filter optimization messes up when a nested select is present in the same block

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13630304#comment-13630304
 ] 

Simon Helsen commented on JENA-432:
---

I can see that, but I am not sure what you're trying to say. Do we agree that 
something is broken? I never said that adding the block curlies as a workaround 
is a strange fix, i.e. that it is a surprise the optimizer can handle that 
case. The point is that the block should not be necessary. I was just giving 
the workaround as an indicator the assignment optimization does not run into 
trouble when the block is present

> Filter optimization messes up when a nested select is present in the same 
> block
> ---
>
> Key: JENA-432
> URL: https://issues.apache.org/jira/browse/JENA-432
> Project: Apache Jena
>  Issue Type: Bug
>  Components: ARQ
>Affects Versions: Jena 2.10.0
>Reporter: Simon Helsen
>
> The following test query:
> SELECT ?test ?s1
> WHERE {
>   ?test ?p1 ?o1.
>   FILTER ( ?test =  || ?test =  )  
>   OPTIONAL {
> SELECT ?s1
> { ?s1 ?p2 ?o2 }
>   }
> }
> produces the following plan:
> (project (?test ?s1)
>   (disjunction
> (table empty)
> (table empty)))
> Something goes wrong with the FILTER expansion. As a workaround, we observed 
> that the following variation:
> SELECT ?test ?s1
> WHERE {
>   { ?test ?p1 ?o1.
> FILTER ( ?test =  || ?test =  ) 
>  
>   }
>   OPTIONAL {
> SELECT ?s1
> { ?s1 ?p2 ?o2 }
>   }
> }
> produces the correct plan:
> (project (?test ?s1)
>   (leftjoin
> (disjunction
>   (assign ((?test ))
> (bgp (triple  ?p1 ?o1)))
>   (assign ((?test ))
> (bgp (triple  ?p1 ?o1
> (project (?s1)
>   (bgp (triple ?s1 ?/p2 ?/o2)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Comment Edited] (JENA-432) Filter optimization messes up when a nested select is present in the same block

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13630213#comment-13630213
 ] 

Simon Helsen edited comment on JENA-432 at 4/12/13 3:53 PM:


no, the following query

SELECT ?test ?p1
WHERE {
  ?test ?p1 ?s2.
  FILTER ( ?test =  || ?test =  )
  { SELECT ?s1
{ ?s2 ?p2 ?o2 }
  }
} 

may have a more realistic flavor to it, i.e. you do something in a sub query, 
use that result in a bgp which is further restricted using a FILTER with an || 
assignment pattern (so the optimizer will attempt to expand the assignments). 
It still produces the following bogus plan:

(project (?test ?s1)
  (disjunction
(table empty)
(table empty)))


  was (Author: shelsen):
no, the following query

SELECT ?test ?p1
WHERE {
  ?test ?p1 ?s2.
  FILTER ( ?test =  || ?test =  )
  SELECT ?s1
{ ?s2 ?p2 ?o2 }
} 

may have a more realistic flavor to it, i.e. you do something in a sub query, 
use that result in a bgp which is further restricted using a FILTER with an || 
assignment pattern (so the optimizer will attempt to expand the assignments). 
It still produces the following bogus plan:

(project (?test ?s1)
  (disjunction
(table empty)
(table empty)))

  
> Filter optimization messes up when a nested select is present in the same 
> block
> ---
>
> Key: JENA-432
> URL: https://issues.apache.org/jira/browse/JENA-432
> Project: Apache Jena
>  Issue Type: Bug
>  Components: ARQ
>Affects Versions: Jena 2.10.0
>Reporter: Simon Helsen
>
> The following test query:
> SELECT ?test ?s1
> WHERE {
>   ?test ?p1 ?o1.
>   FILTER ( ?test =  || ?test =  )  
>   OPTIONAL {
> SELECT ?s1
> { ?s1 ?p2 ?o2 }
>   }
> }
> produces the following plan:
> (project (?test ?s1)
>   (disjunction
> (table empty)
> (table empty)))
> Something goes wrong with the FILTER expansion. As a workaround, we observed 
> that the following variation:
> SELECT ?test ?s1
> WHERE {
>   { ?test ?p1 ?o1.
> FILTER ( ?test =  || ?test =  ) 
>  
>   }
>   OPTIONAL {
> SELECT ?s1
> { ?s1 ?p2 ?o2 }
>   }
> }
> produces the correct plan:
> (project (?test ?s1)
>   (leftjoin
> (disjunction
>   (assign ((?test ))
> (bgp (triple  ?p1 ?o1)))
>   (assign ((?test ))
> (bgp (triple  ?p1 ?o1
> (project (?s1)
>   (bgp (triple ?s1 ?/p2 ?/o2)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-432) Filter optimization messes up when a nested select is present in the same block

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13630213#comment-13630213
 ] 

Simon Helsen commented on JENA-432:
---

no, the following query

SELECT ?test ?p1
WHERE {
  ?test ?p1 ?s2.
  FILTER ( ?test =  || ?test =  )
  SELECT ?s1
{ ?s2 ?p2 ?o2 }
} 

may have a more realistic flavor to it, i.e. you do something in a sub query, 
use that result in a bgp which is further restricted using a FILTER with an || 
assignment pattern (so the optimizer will attempt to expand the assignments). 
It still produces the following bogus plan:

(project (?test ?s1)
  (disjunction
(table empty)
(table empty)))


> Filter optimization messes up when a nested select is present in the same 
> block
> ---
>
> Key: JENA-432
> URL: https://issues.apache.org/jira/browse/JENA-432
> Project: Apache Jena
>  Issue Type: Bug
>  Components: ARQ
>Affects Versions: Jena 2.10.0
>Reporter: Simon Helsen
>
> The following test query:
> SELECT ?test ?s1
> WHERE {
>   ?test ?p1 ?o1.
>   FILTER ( ?test =  || ?test =  )  
>   OPTIONAL {
> SELECT ?s1
> { ?s1 ?p2 ?o2 }
>   }
> }
> produces the following plan:
> (project (?test ?s1)
>   (disjunction
> (table empty)
> (table empty)))
> Something goes wrong with the FILTER expansion. As a workaround, we observed 
> that the following variation:
> SELECT ?test ?s1
> WHERE {
>   { ?test ?p1 ?o1.
> FILTER ( ?test =  || ?test =  ) 
>  
>   }
>   OPTIONAL {
> SELECT ?s1
> { ?s1 ?p2 ?o2 }
>   }
> }
> produces the correct plan:
> (project (?test ?s1)
>   (leftjoin
> (disjunction
>   (assign ((?test ))
> (bgp (triple  ?p1 ?o1)))
>   (assign ((?test ))
> (bgp (triple  ?p1 ?o1
> (project (?s1)
>   (bgp (triple ?s1 ?/p2 ?/o2)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-432) Filter optimization messes up when a nested select is present in the same block

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13630175#comment-13630175
 ] 

Simon Helsen commented on JENA-432:
---

I realize the scope is different, but I don't see why the query is semantically 
different. There is nothing in the filter clause which refers to anything in 
the second part of the query. Besides, if you comment out the FILTER 
constraint, you get again a plan which makes sense (even without the block)

I agree the query is contrived, but the original was way too complex to show 
here and also shows proprietary stuff. But I don't see why it matters. The 
following plan:

(project (?test ?s1)
  (disjunction
(table empty)
(table empty)))

is never correct.

> Filter optimization messes up when a nested select is present in the same 
> block
> ---
>
> Key: JENA-432
> URL: https://issues.apache.org/jira/browse/JENA-432
> Project: Apache Jena
>  Issue Type: Bug
>  Components: ARQ
>Affects Versions: Jena 2.10.0
>Reporter: Simon Helsen
>
> The following test query:
> SELECT ?test ?s1
> WHERE {
>   ?test ?p1 ?o1.
>   FILTER ( ?test =  || ?test =  )  
>   OPTIONAL {
> SELECT ?s1
> { ?s1 ?p2 ?o2 }
>   }
> }
> produces the following plan:
> (project (?test ?s1)
>   (disjunction
> (table empty)
> (table empty)))
> Something goes wrong with the FILTER expansion. As a workaround, we observed 
> that the following variation:
> SELECT ?test ?s1
> WHERE {
>   { ?test ?p1 ?o1.
> FILTER ( ?test =  || ?test =  ) 
>  
>   }
>   OPTIONAL {
> SELECT ?s1
> { ?s1 ?p2 ?o2 }
>   }
> }
> produces the correct plan:
> (project (?test ?s1)
>   (leftjoin
> (disjunction
>   (assign ((?test ))
> (bgp (triple  ?p1 ?o1)))
>   (assign ((?test ))
> (bgp (triple  ?p1 ?o1
> (project (?s1)
>   (bgp (triple ?s1 ?/p2 ?/o2)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (JENA-432) Filter optimization messes up when a nested select is present in the same block

[Simon Helsen (JIRA)]

Simon Helsen created JENA-432:
-

 Summary: Filter optimization messes up when a nested select is 
present in the same block
 Key: JENA-432
 URL: https://issues.apache.org/jira/browse/JENA-432
 Project: Apache Jena
  Issue Type: Bug
  Components: ARQ
Affects Versions: Jena 2.10.0
Reporter: Simon Helsen


The following test query:

SELECT ?test ?s1
WHERE {
  ?test ?p1 ?o1.
  FILTER ( ?test =  || ?test =  )  
  OPTIONAL {
SELECT ?s1
{ ?s1 ?p2 ?o2 }
  }
}

produces the following plan:

(project (?test ?s1)
  (disjunction
(table empty)
(table empty)))

Something goes wrong with the FILTER expansion. As a workaround, we observed 
that the following variation:

SELECT ?test ?s1
WHERE {
  { ?test ?p1 ?o1.
FILTER ( ?test =  || ?test =  )  
  }
  OPTIONAL {
SELECT ?s1
{ ?s1 ?p2 ?o2 }
  }
}

produces the correct plan:

(project (?test ?s1)
  (leftjoin
(disjunction
  (assign ((?test ))
(bgp (triple  ?p1 ?o1)))
  (assign ((?test ))
(bgp (triple  ?p1 ?o1
(project (?s1)
  (bgp (triple ?s1 ?/p2 ?/o2)


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (JENA-408) SDB should not attempt to execute multiple update statements in one jdbc statement

[Simon Helsen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/JENA-408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Simon Helsen updated JENA-408:
--

Attachment: patch.txt

> SDB should not attempt to execute multiple update statements in one jdbc 
> statement
> --
>
> Key: JENA-408
> URL: https://issues.apache.org/jira/browse/JENA-408
> Project: Apache Jena
>  Issue Type: Bug
>  Components: SDB
>Affects Versions: SDB 1.3.6
>Reporter: Simon Helsen
> Fix For: SDB 1.3.6
>
> Attachments: patch.txt
>
>
> Some of the db providers (I noticed in db2, but there may be others) attempt 
> to execute update statements by separating them with a semi-colon. This is 
> not supported by the jdbc spec and most jdbc drivers won't be able to handle 
> this correctly. 
> I am not entirely sure why such statements are present in SDB but I suspect 
> that old jdbc 3 drivers for db2 permitted this. The fix is relatively simple 
> and a patch is being attached

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (JENA-408) SDB should not attempt to execute multiple update statements in one jdbc statement

[Simon Helsen (JIRA)]

Simon Helsen created JENA-408:
-

 Summary: SDB should not attempt to execute multiple update 
statements in one jdbc statement
 Key: JENA-408
 URL: https://issues.apache.org/jira/browse/JENA-408
 Project: Apache Jena
  Issue Type: Bug
  Components: SDB
Affects Versions: SDB 1.3.6
Reporter: Simon Helsen
 Fix For: SDB 1.3.6
 Attachments: patch.txt

Some of the db providers (I noticed in db2, but there may be others) attempt to 
execute update statements by separating them with a semi-colon. This is not 
supported by the jdbc spec and most jdbc drivers won't be able to handle this 
correctly. 

I am not entirely sure why such statements are present in SDB but I suspect 
that old jdbc 3 drivers for db2 permitted this. The fix is relatively simple 
and a patch is being attached

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-407) toLowerCase without Locale.English causing trouble in some language regions (Turkey especially)

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13590035#comment-13590035
 ] 

Simon Helsen commented on JENA-407:
---

thanks Andy. Great

> toLowerCase without Locale.English causing trouble in some language regions 
> (Turkey especially)
> ---
>
> Key: JENA-407
> URL: https://issues.apache.org/jira/browse/JENA-407
> Project: Apache Jena
>  Issue Type: Bug
>  Components: Jena
>Affects Versions: Jena 2.10.0
>Reporter: Simon Helsen
>Assignee: Andy Seaborne
>Priority: Minor
> Fix For: Jena 2.10.1
>
>
> The instance I am referring to concretely is the language tag constructor: 
> LanguageTag. 
> It makes the following call on line 41:  String lc = tag.toLowerCase(); This 
> should be corrected to  String lc = tag.toLowerCase(Locale.English);
> The problem is that otherwise, it use the machine default language to produce 
> the lower cases which in some Locales (Turkey being one of them) incorrectly 
> lowercases letters like 'I'. Because the tag is a 'technical' term (not an 
> actual piece of language) it should lowercase in English
> The effect of this particular instance is that we see 
> System.err.println("Internal Error in static initializer of IanaLnaguageTag.")
> appear in std.err and it has raised concerns with our customers.
> In general, any occurrence of toLowerCase should be adjusted if it lowercases 
> a technical term.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (JENA-407) toLowerCase without Locale.English causing trouble in some language regions (Turkey especially)

[Simon Helsen (JIRA)]

Simon Helsen created JENA-407:
-

 Summary: toLowerCase without Locale.English causing trouble in 
some language regions (Turkey especially)
 Key: JENA-407
 URL: https://issues.apache.org/jira/browse/JENA-407
 Project: Apache Jena
  Issue Type: Bug
  Components: ARQ
Affects Versions: Jena 2.10.0
Reporter: Simon Helsen
Priority: Minor


The instance I am referring to concretely is the language tag constructor: 
LanguageTag. 

It makes the following call on line 41:  String lc = tag.toLowerCase(); This 
should be corrected to  String lc = tag.toLowerCase(Locale.English);

The problem is that otherwise, it use the machine default language to produce 
the lower cases which in some Locales (Turkey being one of them) incorrectly 
lowercases letters like 'I'. Because the tag is a 'technical' term (not an 
actual piece of language) it should lowercase in English

The effect of this particular instance is that we see 

System.err.println("Internal Error in static initializer of IanaLnaguageTag.")

appear in std.err and it has raised concerns with our customers.

In general, any occurrence of toLowerCase should be adjusted if it lowercases a 
technical term.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-289) Respect query timeouts in TDB implementation

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13564329#comment-13564329
 ] 

Simon Helsen commented on JENA-289:
---

Andy, can you elaborate the recent changes? Is this a change in the graph 
matching algorithm? 

> Respect query timeouts in TDB implementation
> 
>
> Key: JENA-289
> URL: https://issues.apache.org/jira/browse/JENA-289
> Project: Apache Jena
>  Issue Type: Improvement
>  Components: TDB
>Affects Versions: TDB 0.9.1
>Reporter: Mark Buquor
>Assignee: Andy Seaborne
> Fix For: TDB 0.10.0
>
> Attachments: TestTimeout.java, TestTimeout.java
>
>
> In general use, we sometimes see queries throw QueryCancelledException 
> several seconds/minutes after the expected timeout. This is acceptable to a 
> degree, but it appears that there are cases where a rogue query could execute 
> unmitigated. The attached testcase is an example of a query that will execute 
> and consume CPU/heap until an OutOfMemoryError is thrown.
> Example: The following query with 10s timeouts executed for ~7 minutes before 
> throwing an OOME.
> Aug 3, 2012 10:18:19 AM Executing query [limit=1,000 timeout1=10s 
> timeout2=10s]: SELECT * WHERE { ?a ?b ?c . ?c ?d ?e }
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.util.HashSet.(HashSet.java:86)
> at org.openjena.atlas.iterator.FilterUnique.(FilterUnique.java:26)
> at org.openjena.atlas.iterator.Iter.distinct(Iter.java:438)
> at 
> com.hp.hpl.jena.tdb.solver.StageMatchTuple.makeNextStage(StageMatchTuple.java:116)
> at 
> com.hp.hpl.jena.tdb.solver.StageMatchTuple.makeNextStage(StageMatchTuple.java:44)
> at 
> org.openjena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:49)
> at org.openjena.atlas.iterator.Iter$4.hasNext(Iter.java:295)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIterPlainWrapper.hasNextBinding(QueryIterPlainWrapper.java:54)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIterSlice.hasNextBinding(QueryIterSlice.java:76)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorWrapper.hasNextBinding(QueryIteratorWrapper.java:40)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorWrapper.hasNextBinding(QueryIteratorWrapper.java:40)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorWrapper.hasNextBinding(QueryIteratorWrapper.java:40)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
> at 
> com.hp.hpl.jena.sparql.engine.ResultSetStream.hasNext(ResultSetStream.java:72)
> at 
> com.hp.hpl.jena.sparql.resultset.ResultSetApply.apply(ResultSetApply.java:41)
> at com.hp.hpl.jena.sparql.resultset.XMLOutput.format(XMLOutput.java:52)
> at 
> com.hp.hpl.jena.query.ResultSetFormatter.outputAsXML(ResultSetFormatter.java:482)
> at 
> com.hp.hpl.jena.query.ResultSetFormatter.outputAsXML(ResultSetFormatter.java:460)
> at TestTimout.main(TestTimout.java:84)
> Aug 3, 2012 10:25:41 AM Finished

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-385) NPE during abort

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559774#comment-13559774
 ] 

Simon Helsen commented on JENA-385:
---

You mean StoreConnection.release() ? I am not sure what you're suggesting when 
you say "TDB does not close files unless asked to". The way I read the stack 
trace, it looks like TDB wants to write/sync to a file which has been closed 
unexpectedly. We have no code doing that sort of thing, so I presume a network 
or file system glitch could have closed the file under the hood

> NPE during abort
> 
>
> Key: JENA-385
> URL: https://issues.apache.org/jira/browse/JENA-385
> Project: Apache Jena
>  Issue Type: Bug
>  Components: TDB
>Affects Versions: TDB 0.9.4
>Reporter: Simon Helsen
>Assignee: Andy Seaborne
>
> we ran into a non-reproducible glitch where a transaction was unable to 
> commit (we don't know exactly why - could be a network glitch or hard drive 
> hickup). As a consequence, an abort was initiated which let to an NPE because 
> the journalObjfile was already null. It happens in the following code in 
> NodeTableTrans on the call to truncate. 
> public void abort(Transaction txn)
> {
> debug("abort") ;
> if ( nodeTableJournal == null )
> throw new TDBTransactionException(txn.getLabel()+": Not in a 
> transaction for a commit to happen") ;
> // Ensure the cache does not flush.
> nodeTableJournal = null ;
> // then make sure the journal file is empty.
> journalObjFile.truncate(journalObjFileStartOffset) ;
> journalObjFile.sync() ;
> finish() ;
> }
> Should there not just be a check here to verify if journalObjFile != null? So
> public void abort(Transaction txn)
> {
> debug("abort") ;
> if ( nodeTableJournal == null )
> throw new TDBTransactionException(txn.getLabel()+": Not in a 
> transaction for a commit to happen") ;
> // Ensure the cache does not flush.
> nodeTableJournal = null ;
> // then make sure the journal file is empty.
> if (journalObjFile != null) {
>journalObjFile.truncate(journalObjFileStartOffset) ;
>journalObjFile.sync() ;
> }
> finish() ;
> }

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-385) NPE during abort

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559746#comment-13559746
 ] 

Simon Helsen commented on JENA-385:
---

ok, well I agree the NPE is only a consequence. but why would the db be suspect 
at this point? This is the original problem (after which we attempted to abort)

Caused by: com.hp.hpl.jena.tdb.transaction.TDBTransactionException: Abort 
during prepare - transaction did not commit
at 
com.hp.hpl.jena.tdb.transaction.Transaction.commit(Transaction.java:130)
at 
com.hp.hpl.jena.tdb.transaction.DatasetGraphTxn.commit(DatasetGraphTxn.java:44)
at 
com.hp.hpl.jena.tdb.transaction.DatasetGraphTransaction._commit(DatasetGraphTransaction.java:157)
at 
com.hp.hpl.jena.sparql.core.DatasetGraphTrackActive.commit(DatasetGraphTrackActive.java:54)
at com.hp.hpl.jena.sparql.core.DatasetImpl.commit(DatasetImpl.java:141)
at 
com.ibm.team.jfs.rdf.internal.jena.tdb.JenaTxTdbProvider.storeWriteOperation(JenaTxTdbProvider.java:2322)
Caused by: com.hp.hpl.jena.tdb.base.file.FileException: FileBase.sync
at com.hp.hpl.jena.tdb.base.file.FileBase.sync(FileBase.java:110)
at 
com.hp.hpl.jena.tdb.base.file.BufferChannelFile.sync(BufferChannelFile.java:147)
at 
com.hp.hpl.jena.tdb.base.objectfile.ObjectFileStorage.sync(ObjectFileStorage.java:379)
at 
com.hp.hpl.jena.tdb.nodetable.NodeTableNative.sync(NodeTableNative.java:251)
at 
com.hp.hpl.jena.tdb.nodetable.NodeTableCache.sync(NodeTableCache.java:219)
at 
com.hp.hpl.jena.tdb.nodetable.NodeTableWrapper.sync(NodeTableWrapper.java:75)
at 
com.hp.hpl.jena.tdb.transaction.NodeTableTrans.writeNodeJournal(NodeTableTrans.java:306)
at 
com.hp.hpl.jena.tdb.transaction.NodeTableTrans.commitPrepare(NodeTableTrans.java:271)
at 
com.hp.hpl.jena.tdb.transaction.Transaction.prepare(Transaction.java:185)
at 
com.hp.hpl.jena.tdb.transaction.Transaction.commit(Transaction.java:115)
Caused by: java.nio.channels.ClosedChannelException
at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:100)
at sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:354)
at com.hp.hpl.jena.tdb.base.file.FileBase.sync(FileBase.java:108)



> NPE during abort
> 
>
> Key: JENA-385
> URL: https://issues.apache.org/jira/browse/JENA-385
> Project: Apache Jena
>  Issue Type: Bug
>  Components: TDB
>Affects Versions: TDB 0.9.4
>Reporter: Simon Helsen
>Assignee: Andy Seaborne
>
> we ran into a non-reproducible glitch where a transaction was unable to 
> commit (we don't know exactly why - could be a network glitch or hard drive 
> hickup). As a consequence, an abort was initiated which let to an NPE because 
> the journalObjfile was already null. It happens in the following code in 
> NodeTableTrans on the call to truncate. 
> public void abort(Transaction txn)
> {
> debug("abort") ;
> if ( nodeTableJournal == null )
> throw new TDBTransactionException(txn.getLabel()+": Not in a 
> transaction for a commit to happen") ;
> // Ensure the cache does not flush.
> nodeTableJournal = null ;
> // then make sure the journal file is empty.
> journalObjFile.truncate(journalObjFileStartOffset) ;
> journalObjFile.sync() ;
> finish() ;
> }
> Should there not just be a check here to verify if journalObjFile != null? So
> public void abort(Transaction txn)
> {
> debug("abort") ;
> if ( nodeTableJournal == null )
> throw new TDBTransactionException(txn.getLabel()+": Not in a 
> transaction for a commit to happen") ;
> // Ensure the cache does not flush.
> nodeTableJournal = null ;
> // then make sure the journal file is empty.
> if (journalObjFile != null) {
>journalObjFile.truncate(journalObjFileStartOffset) ;
>journalObjFile.sync() ;
> }
> finish() ;
> }

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (JENA-385) NPE during abort

[Simon Helsen (JIRA)]

Simon Helsen created JENA-385:
-

 Summary: NPE during abort
 Key: JENA-385
 URL: https://issues.apache.org/jira/browse/JENA-385
 Project: Apache Jena
  Issue Type: Bug
  Components: TDB
Affects Versions: TDB 0.9.4
Reporter: Simon Helsen


we ran into a non-reproducible glitch where a transaction was unable to commit 
(we don't know exactly why - could be a network glitch or hard drive hickup). 
As a consequence, an abort was initiated which let to an NPE because the 
journalObjfile was already null. It happens in the following code in 
NodeTableTrans on the call to truncate. 

public void abort(Transaction txn)
{
debug("abort") ;
if ( nodeTableJournal == null )
throw new TDBTransactionException(txn.getLabel()+": Not in a 
transaction for a commit to happen") ;
// Ensure the cache does not flush.
nodeTableJournal = null ;
// then make sure the journal file is empty.
journalObjFile.truncate(journalObjFileStartOffset) ;
journalObjFile.sync() ;
finish() ;
}

Should there not just be a check here to verify if journalObjFile != null? So

public void abort(Transaction txn)
{
debug("abort") ;
if ( nodeTableJournal == null )
throw new TDBTransactionException(txn.getLabel()+": Not in a 
transaction for a commit to happen") ;
// Ensure the cache does not flush.
nodeTableJournal = null ;
// then make sure the journal file is empty.
if (journalObjFile != null) {
   journalObjFile.truncate(journalObjFileStartOffset) ;
   journalObjFile.sync() ;
}
finish() ;
}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-289) Respect query timeouts in TDB implementation

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13479184#comment-13479184
 ] 

Simon Helsen commented on JENA-289:
---

We execute all our queries with defaultUnionGraph=true. The query in trouble is 
quite ugly and has the following form, but I had to obfuscate it:

PREFIX pr1:   
PREFIX rdf:  
PREFIX pr2:  
PREFIX pr3:  
PREFIX pr4:  
SELECT DISTINCT ?R1 ?R1_resourceContext ?R1_v1 ?R1_v2 ?R1_v3 ?R1_v4 ?R1_v5 
?R1_v6 ?R1_v7 ?R1_v8 ?R1_v9 ?R1_v10 ?R1_v11
WHERE
{ { ?R1 pr4:pred1   }
OPTIONAL
{ ?R1 pr2:pred2 ?R1_v7 }
OPTIONAL
{ ?R1 pr1:pred3 ?R1_v10 }
OPTIONAL
{ ?R1 pr1:pred4 ?R1_v6 }
OPTIONAL
{ ?R1 pr1:pred5 ?R1_v11 }
OPTIONAL
{ ?R1 pr1:pred6 ?R1_v8 }
OPTIONAL
{ ?R1 pr3:pred7 ?R1_v12 }
FILTER ( ! bound(?R1_v12) )
OPTIONAL
{ ?R1 pr3:pred8 ?R1_v1 .
?R1_v1 pr1:pred6 ?R1_uv2
}
OPTIONAL
{ ?R1 pr3:pred9 ?R1_v2 }
OPTIONAL
{ { SELECT ?R1 (count(?b) AS ?R1_v9)
WHERE
{ ?R1 pr4:pred1  .
?b pr3:pred10 ?R1
}
GROUP BY ?R1
OFFSET 35640
LIMIT 20
}
}
OPTIONAL
{ ?R1 pr3:pred11 ?R1_v5 }
OPTIONAL
{ ?R1 pr3:pred12 ?R1_v4 }
OPTIONAL
{ ?R1 pr4:pred1 ?R1_v3 .
?R1_v3 rdf:value ?R1_uv1
}
?R1 rdf:type pr3:pred13 .
?R1 pr2:pred14 ?R1_resourceContext
}
OFFSET 35640
LIMIT 20

Note: this comes from our client and they acknowledge it is bad. But as we 
noted elsewhere in the issue, one bad query from one client can bring the 
server on its knees and one of the main use-cases for abort is to prevent a 
rogue query from continuing. Note that we also run into OMEs with a very large 
heap. This is either caused by the same reason you mention in an earlier 
comment (the union default graph needs to suppress duplicates) or, because this 
is executed in a transaction, all subsequent write activity is kept in memory 
eventually exploding it

> Respect query timeouts in TDB implementation
> 
>
> Key: JENA-289
> URL: https://issues.apache.org/jira/browse/JENA-289
> Project: Apache Jena
>  Issue Type: Improvement
>  Components: TDB
>Affects Versions: TDB 0.9.1
>Reporter: Mark Buquor
> Attachments: TestTimeout.java, TestTimeout.java
>
>
> In general use, we sometimes see queries throw QueryCancelledException 
> several seconds/minutes after the expected timeout. This is acceptable to a 
> degree, but it appears that there are cases where a rogue query could execute 
> unmitigated. The attached testcase is an example of a query that will execute 
> and consume CPU/heap until an OutOfMemoryError is thrown.
> Example: The following query with 10s timeouts executed for ~7 minutes before 
> throwing an OOME.
> Aug 3, 2012 10:18:19 AM Executing query [limit=1,000 timeout1=10s 
> timeout2=10s]: SELECT * WHERE { ?a ?b ?c . ?c ?d ?e }
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.util.HashSet.(HashSet.java:86)
> at org.openjena.atlas.iterator.FilterUnique.(FilterUnique.java:26)
> at org.openjena.atlas.iterator.Iter.distinct(Iter.java:438)
> at 
> com.hp.hpl.jena.tdb.solver.StageMatchTuple.makeNextStage(StageMatchTuple.java:116)
> at 
> com.hp.hpl.jena.tdb.solver.StageMatchTuple.makeNextStage(StageMatchTuple.java:44)
> at 
> org.openjena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:49)
> at org.openjena.atlas.iterator.Iter$4.hasNext(Iter.java:295)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIterPlainWrapper.hasNextBinding(QueryIterPlainWrapper.java:54)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIterSlice.hasNextBinding(QueryIterSlice.java:76)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorWrapper.hasNextBinding(QueryIteratorWrapper.java:40)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorWrapper.hasNextBinding(QueryIteratorWrapper.java:40)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorWrapper.hasNextBinding(QueryIteratorWrapper.java:40)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
> at 
> com.hp.hpl.jena.sparql.engine.ResultSetStream.hasNext(ResultSetStream.java:72)
> at 
> com.hp.hpl.jena.sparql.resultset.ResultSetApply.apply(ResultSetApp

[jira] [Commented] (JENA-289) Respect query timeouts in TDB implementation

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13479126#comment-13479126
 ] 

Simon Helsen commented on JENA-289:
---

actually, we dug deeper into the trouble query and it appears that in 
QueryExecutionBase, the queryIterator remains null for a very long time, which 
is essentially what this bug is addressing. The query is nasty and is returning 
too many results, so I told them this needs to be fixed. Now, I am trying to 
understand the code better. Obviously, plan creation is fast, by does it take 
so long to get the iterator object first? looking at the code path, I can't see 
where the time goes

> Respect query timeouts in TDB implementation
> 
>
> Key: JENA-289
> URL: https://issues.apache.org/jira/browse/JENA-289
> Project: Apache Jena
>  Issue Type: Improvement
>  Components: TDB
>Affects Versions: TDB 0.9.1
>Reporter: Mark Buquor
> Attachments: TestTimeout.java, TestTimeout.java
>
>
> In general use, we sometimes see queries throw QueryCancelledException 
> several seconds/minutes after the expected timeout. This is acceptable to a 
> degree, but it appears that there are cases where a rogue query could execute 
> unmitigated. The attached testcase is an example of a query that will execute 
> and consume CPU/heap until an OutOfMemoryError is thrown.
> Example: The following query with 10s timeouts executed for ~7 minutes before 
> throwing an OOME.
> Aug 3, 2012 10:18:19 AM Executing query [limit=1,000 timeout1=10s 
> timeout2=10s]: SELECT * WHERE { ?a ?b ?c . ?c ?d ?e }
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.util.HashSet.(HashSet.java:86)
> at org.openjena.atlas.iterator.FilterUnique.(FilterUnique.java:26)
> at org.openjena.atlas.iterator.Iter.distinct(Iter.java:438)
> at 
> com.hp.hpl.jena.tdb.solver.StageMatchTuple.makeNextStage(StageMatchTuple.java:116)
> at 
> com.hp.hpl.jena.tdb.solver.StageMatchTuple.makeNextStage(StageMatchTuple.java:44)
> at 
> org.openjena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:49)
> at org.openjena.atlas.iterator.Iter$4.hasNext(Iter.java:295)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIterPlainWrapper.hasNextBinding(QueryIterPlainWrapper.java:54)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIterSlice.hasNextBinding(QueryIterSlice.java:76)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorWrapper.hasNextBinding(QueryIteratorWrapper.java:40)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorWrapper.hasNextBinding(QueryIteratorWrapper.java:40)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorWrapper.hasNextBinding(QueryIteratorWrapper.java:40)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
> at 
> com.hp.hpl.jena.sparql.engine.ResultSetStream.hasNext(ResultSetStream.java:72)
> at 
> com.hp.hpl.jena.sparql.resultset.ResultSetApply.apply(ResultSetApply.java:41)
> at com.hp.hpl.jena.sparql.resultset.XMLOutput.format(XMLOutput.java:52)
> at 
> com.hp.hpl.jena.query.ResultSetFormatter.outputAsXML(ResultSetFormatter.java:482)
> at 
> com.hp.hpl.jena.query.ResultSetFormatter.outputAsXML(ResultSetFormatter.java:460)
> at TestTimout.main(TestTimout.java:84)
> Aug 3, 2012 10:25:41 AM Finished

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-289) Respect query timeouts in TDB implementation

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478239#comment-13478239
 ] 

Simon Helsen commented on JENA-289:
---

I have a question about the abort and the problem reported here. Right now, we 
are looking into a workaround that if a bad query occurs and abort is 
requested, we just keep requesting the abort (like every so many seconds) until 
the abort works. The reason is that from the explanation above, it seems that 
the abort only doesn't catch until the first result is returned (i.e. the 
iterator starts). Now, we have a instance of an expensive query which we try to 
abort after 30 seconds. That doesn't seem to work, so I keep calling 
QueryExecution.abort() every 5 seconds. Yet, the query keeps going and after 16 
minutes, we hit the OME because the journal is exploding.  

I can't seem to find in the code why ARQ would not keep trying to abort if it 
is requested more than once. We will try to examine where the query is hanging 
during these 16 minutes, but my question was mostly to try to understand if our 
strategy to keep requesting abort() on the QueryExecution (in a separate thread 
of course), would eventually have to work

> Respect query timeouts in TDB implementation
> 
>
> Key: JENA-289
> URL: https://issues.apache.org/jira/browse/JENA-289
> Project: Apache Jena
>  Issue Type: Improvement
>  Components: TDB
>Affects Versions: TDB 0.9.1
>Reporter: Mark Buquor
> Attachments: TestTimeout.java, TestTimeout.java
>
>
> In general use, we sometimes see queries throw QueryCancelledException 
> several seconds/minutes after the expected timeout. This is acceptable to a 
> degree, but it appears that there are cases where a rogue query could execute 
> unmitigated. The attached testcase is an example of a query that will execute 
> and consume CPU/heap until an OutOfMemoryError is thrown.
> Example: The following query with 10s timeouts executed for ~7 minutes before 
> throwing an OOME.
> Aug 3, 2012 10:18:19 AM Executing query [limit=1,000 timeout1=10s 
> timeout2=10s]: SELECT * WHERE { ?a ?b ?c . ?c ?d ?e }
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.util.HashSet.(HashSet.java:86)
> at org.openjena.atlas.iterator.FilterUnique.(FilterUnique.java:26)
> at org.openjena.atlas.iterator.Iter.distinct(Iter.java:438)
> at 
> com.hp.hpl.jena.tdb.solver.StageMatchTuple.makeNextStage(StageMatchTuple.java:116)
> at 
> com.hp.hpl.jena.tdb.solver.StageMatchTuple.makeNextStage(StageMatchTuple.java:44)
> at 
> org.openjena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:49)
> at org.openjena.atlas.iterator.Iter$4.hasNext(Iter.java:295)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIterPlainWrapper.hasNextBinding(QueryIterPlainWrapper.java:54)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIterSlice.hasNextBinding(QueryIterSlice.java:76)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorWrapper.hasNextBinding(QueryIteratorWrapper.java:40)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorWrapper.hasNextBinding(QueryIteratorWrapper.java:40)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorWrapper.hasNextBinding(QueryIteratorWrapper.java:40)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
> at 
> com.hp.hpl.jena.sparql.engine.ResultSetStream.hasNext(ResultSetStream.java:72)
> at 
> com.hp.hpl.jena.sparql.resultset.ResultSetApply.apply(ResultSetApply.java:41)
> at com.hp.hpl.jena.sparql.resultset.XMLOutput.format(XMLOutput.java:52)
> at 
> com.hp.hpl.jena.query.ResultSetFormatter.outputAsXML(ResultSetFormatter.java:482)
> at 
> com.hp.hpl.jena.query.ResultSetFormatter.outputAsXML(ResultSetFormatter.java:460)
> at TestTimout.main(TestTimout.java:84)
> Aug 3, 2012 10:25:41 AM Finished

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-336) Fuseki assembly-server.xml doesn't build in Windows

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478210#comment-13478210
 ] 

Simon Helsen commented on JENA-336:
---

ah, that looks like the problem I was seeing as well

> Fuseki assembly-server.xml doesn't build in Windows
> ---
>
> Key: JENA-336
> URL: https://issues.apache.org/jira/browse/JENA-336
> Project: Apache Jena
>  Issue Type: Bug
>  Components: Fuseki
>Reporter: Stephen Allen
>Assignee: Stephen Allen
>
> There is a regex in Fuseki's assembly-server.xml that does not work in 
> Windows.  Doing a build of Fuseki fails with an error [1].
> It appears to be line 45:  %regex[.*/]
> I'm guessing this has something to do with Windows changing / to \ and the 
> regex parser thinking that is an escape character.
> Can I just comment this line out?  A note indicates that this is just to 
> prevent "skipping" messages.
> [1]
> [INFO] --- maven-assembly-plugin:2.2.1:single (create-server-assembly) @ 
> jena-fuseki ---
> [INFO] Reading assembly descriptor: assembly-server.xml
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-assembly-plugin:2.2.1:single 
> (create-server-assembly) on project jena-fuseki: Execution 
> create-server-assembly of goal 
> org.apache.maven.plugins:maven-assembly-plugin:2.2.1:single failed: 
> Unexpected internal error near index 3
> [ERROR] .*\
> [ERROR] ^
> [ERROR] -> [Help 1]
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute 
> goal org.apache.maven.plugins:maven-assembly-plugin:2.2.1:single 
> (create-server-assembly) on project jena-fuseki: Execution 
> create-server-assembly of goal 
> org.apache.maven.plugins:maven-assembly-plugin:2.2.1:single failed: 
> Unexpected internal error near index 3
> .*\
>^
> at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:225)
> at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
> at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
> at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
> at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
> at 
> org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
> at 
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320)
> at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
> at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
> at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
> at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at 
> org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290)
> at 
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230)
> at 
> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
> at 
> org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
> Caused by: org.apache.maven.plugin.PluginExecutionException: Execution 
> create-server-assembly of goal 
> org.apache.maven.plugins:maven-assembly-plugin:2.2.1:single failed: 
> Unexpected internal error near index 3
> .*\
>^
> at 
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:110)
> at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
> ... 19 more
> Caused by: java.util.regex.PatternSyntaxException: Unexpected internal error 
> near index 3
> .*\
>^
> at java.util.regex.Pattern.error(Pattern.java:1924)
> at java.util.regex.Pattern.compile(Pattern.java:1671)
> at java.util.regex.Pattern.(Pattern.java:1337)
> at java.util.regex.Pattern.compile(Pattern.java:1022)
> at java.util.regex.Pattern.matches(Pattern.java:1128)
> at java.lang.String.matches(String.java:2063)
> at 
> org.codehaus.plexus.util.SelectorUtils.matchPath(SelectorUtils.java:262)
> at 
> org.codehaus.plexus.components.io.fileselectors.IncludeExcludeFileSelector.matchPath(IncludeExcludeFileSelector.java:189)
> at 
> org.codeha

[jira] [Commented] (JENA-289) Respect query timeouts in TDB implementation

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13469701#comment-13469701
 ] 

Simon Helsen commented on JENA-289:
---

ok, I understand now given the comments on the jena dev mailing list. The crude 
workaround is that in 2.7.4, killing the process will not (likely) cause data 
integrity problems

> Respect query timeouts in TDB implementation
> 
>
> Key: JENA-289
> URL: https://issues.apache.org/jira/browse/JENA-289
> Project: Apache Jena
>  Issue Type: Improvement
>  Components: TDB
>Affects Versions: TDB 0.9.1
>Reporter: Mark Buquor
> Attachments: TestTimeout.java, TestTimeout.java
>
>
> In general use, we sometimes see queries throw QueryCancelledException 
> several seconds/minutes after the expected timeout. This is acceptable to a 
> degree, but it appears that there are cases where a rogue query could execute 
> unmitigated. The attached testcase is an example of a query that will execute 
> and consume CPU/heap until an OutOfMemoryError is thrown.
> Example: The following query with 10s timeouts executed for ~7 minutes before 
> throwing an OOME.
> Aug 3, 2012 10:18:19 AM Executing query [limit=1,000 timeout1=10s 
> timeout2=10s]: SELECT * WHERE { ?a ?b ?c . ?c ?d ?e }
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.util.HashSet.(HashSet.java:86)
> at org.openjena.atlas.iterator.FilterUnique.(FilterUnique.java:26)
> at org.openjena.atlas.iterator.Iter.distinct(Iter.java:438)
> at 
> com.hp.hpl.jena.tdb.solver.StageMatchTuple.makeNextStage(StageMatchTuple.java:116)
> at 
> com.hp.hpl.jena.tdb.solver.StageMatchTuple.makeNextStage(StageMatchTuple.java:44)
> at 
> org.openjena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:49)
> at org.openjena.atlas.iterator.Iter$4.hasNext(Iter.java:295)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIterPlainWrapper.hasNextBinding(QueryIterPlainWrapper.java:54)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIterSlice.hasNextBinding(QueryIterSlice.java:76)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorWrapper.hasNextBinding(QueryIteratorWrapper.java:40)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorWrapper.hasNextBinding(QueryIteratorWrapper.java:40)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorWrapper.hasNextBinding(QueryIteratorWrapper.java:40)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
> at 
> com.hp.hpl.jena.sparql.engine.ResultSetStream.hasNext(ResultSetStream.java:72)
> at 
> com.hp.hpl.jena.sparql.resultset.ResultSetApply.apply(ResultSetApply.java:41)
> at com.hp.hpl.jena.sparql.resultset.XMLOutput.format(XMLOutput.java:52)
> at 
> com.hp.hpl.jena.query.ResultSetFormatter.outputAsXML(ResultSetFormatter.java:482)
> at 
> com.hp.hpl.jena.query.ResultSetFormatter.outputAsXML(ResultSetFormatter.java:460)
> at TestTimout.main(TestTimout.java:84)
> Aug 3, 2012 10:25:41 AM Finished

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-289) Respect query timeouts in TDB implementation

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13469678#comment-13469678
 ] 

Simon Helsen commented on JENA-289:
---

Andy, I am not sure I understand what you mean by "crude workaround". 

> Respect query timeouts in TDB implementation
> 
>
> Key: JENA-289
> URL: https://issues.apache.org/jira/browse/JENA-289
> Project: Apache Jena
>  Issue Type: Improvement
>  Components: TDB
>Affects Versions: TDB 0.9.1
>Reporter: Mark Buquor
> Attachments: TestTimeout.java, TestTimeout.java
>
>
> In general use, we sometimes see queries throw QueryCancelledException 
> several seconds/minutes after the expected timeout. This is acceptable to a 
> degree, but it appears that there are cases where a rogue query could execute 
> unmitigated. The attached testcase is an example of a query that will execute 
> and consume CPU/heap until an OutOfMemoryError is thrown.
> Example: The following query with 10s timeouts executed for ~7 minutes before 
> throwing an OOME.
> Aug 3, 2012 10:18:19 AM Executing query [limit=1,000 timeout1=10s 
> timeout2=10s]: SELECT * WHERE { ?a ?b ?c . ?c ?d ?e }
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.util.HashSet.(HashSet.java:86)
> at org.openjena.atlas.iterator.FilterUnique.(FilterUnique.java:26)
> at org.openjena.atlas.iterator.Iter.distinct(Iter.java:438)
> at 
> com.hp.hpl.jena.tdb.solver.StageMatchTuple.makeNextStage(StageMatchTuple.java:116)
> at 
> com.hp.hpl.jena.tdb.solver.StageMatchTuple.makeNextStage(StageMatchTuple.java:44)
> at 
> org.openjena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:49)
> at org.openjena.atlas.iterator.Iter$4.hasNext(Iter.java:295)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIterPlainWrapper.hasNextBinding(QueryIterPlainWrapper.java:54)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIterSlice.hasNextBinding(QueryIterSlice.java:76)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorWrapper.hasNextBinding(QueryIteratorWrapper.java:40)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorWrapper.hasNextBinding(QueryIteratorWrapper.java:40)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorWrapper.hasNextBinding(QueryIteratorWrapper.java:40)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
> at 
> com.hp.hpl.jena.sparql.engine.ResultSetStream.hasNext(ResultSetStream.java:72)
> at 
> com.hp.hpl.jena.sparql.resultset.ResultSetApply.apply(ResultSetApply.java:41)
> at com.hp.hpl.jena.sparql.resultset.XMLOutput.format(XMLOutput.java:52)
> at 
> com.hp.hpl.jena.query.ResultSetFormatter.outputAsXML(ResultSetFormatter.java:482)
> at 
> com.hp.hpl.jena.query.ResultSetFormatter.outputAsXML(ResultSetFormatter.java:460)
> at TestTimout.main(TestTimout.java:84)
> Aug 3, 2012 10:25:41 AM Finished

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Comment Edited] (JENA-289) Respect query timeouts in TDB implementation

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13464029#comment-13464029
 ] 

Simon Helsen edited comment on JENA-289 at 9/27/12 5:48 AM:


The main problem we face is that it may cause a production system to go on its 
knees. We recently had such an instance because of a mistake in how a query was 
written. Without a reliable abort we put ourselves at great risk. And I should 
add that during our testing efforst, we are encountering countless queries 
which run into this situation and refuse to abort. These queries can usually be 
fixed or rewritten, but it seems remarkably easy for our clients to produce 
such queries which require the server to be killed. The latter action then 
often causes corruptions because Jena 2.7.1 is not good at handling abrupt 
process kills. And since there is no 2.7.4 release with all these fixes...

  was (Author: shelsen):
The main problem we face is that may cause a production system to go on its 
knees. We recently had such an instance because of a mistake in how a query was 
written. Without a reliable abort we put ourselves at great risk. 
  
> Respect query timeouts in TDB implementation
> 
>
> Key: JENA-289
> URL: https://issues.apache.org/jira/browse/JENA-289
> Project: Apache Jena
>  Issue Type: Improvement
>  Components: TDB
>Affects Versions: TDB 0.9.1
>Reporter: Mark Buquor
>Priority: Critical
> Attachments: TestTimeout.java, TestTimeout.java
>
>
> In general use, we sometimes see queries throw QueryCancelledException 
> several seconds/minutes after the expected timeout. This is acceptable to a 
> degree, but it appears that there are cases where a rogue query could execute 
> unmitigated. The attached testcase is an example of a query that will execute 
> and consume CPU/heap until an OutOfMemoryError is thrown.
> Example: The following query with 10s timeouts executed for ~7 minutes before 
> throwing an OOME.
> Aug 3, 2012 10:18:19 AM Executing query [limit=1,000 timeout1=10s 
> timeout2=10s]: SELECT * WHERE { ?a ?b ?c . ?c ?d ?e }
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.util.HashSet.(HashSet.java:86)
> at org.openjena.atlas.iterator.FilterUnique.(FilterUnique.java:26)
> at org.openjena.atlas.iterator.Iter.distinct(Iter.java:438)
> at 
> com.hp.hpl.jena.tdb.solver.StageMatchTuple.makeNextStage(StageMatchTuple.java:116)
> at 
> com.hp.hpl.jena.tdb.solver.StageMatchTuple.makeNextStage(StageMatchTuple.java:44)
> at 
> org.openjena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:49)
> at org.openjena.atlas.iterator.Iter$4.hasNext(Iter.java:295)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIterPlainWrapper.hasNextBinding(QueryIterPlainWrapper.java:54)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIterSlice.hasNextBinding(QueryIterSlice.java:76)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorWrapper.hasNextBinding(QueryIteratorWrapper.java:40)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorWrapper.hasNextBinding(QueryIteratorWrapper.java:40)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorWrapper.hasNextBinding(QueryIteratorWrapper.java:40)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
> at 
> com.hp.hpl.jena.sparql.engine.ResultSetStream.hasNext(ResultSetStream.java:72)
> at 
> com.hp.hpl.jena.sparql.resultset.ResultSetApply.apply(ResultSetApply.java:41)
> at com.hp.hpl.jena.sparql.resultset.XMLOutput.format(XMLOutput.java:52)
> at 
> com.hp.hpl.jena.query.ResultSetFormatter.outputAsXML(ResultSetFormatter.java:482)
> at 
> com.hp.hpl.jena.query.ResultSetFormatter.outputAsXML(ResultSetFormatter.java:460)
> at TestTimout.main(TestTimout.java:84)
> Aug 3, 2012 10:25:41 AM Finished

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-289) Respect query timeouts in TDB implementation

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13464029#comment-13464029
 ] 

Simon Helsen commented on JENA-289:
---

The main problem we face is that may cause a production system to go on its 
knees. We recently had such an instance because of a mistake in how a query was 
written. Without a reliable abort we put ourselves at great risk. 

> Respect query timeouts in TDB implementation
> 
>
> Key: JENA-289
> URL: https://issues.apache.org/jira/browse/JENA-289
> Project: Apache Jena
>  Issue Type: Improvement
>  Components: TDB
>Affects Versions: TDB 0.9.1
>Reporter: Mark Buquor
>Priority: Critical
> Attachments: TestTimeout.java, TestTimeout.java
>
>
> In general use, we sometimes see queries throw QueryCancelledException 
> several seconds/minutes after the expected timeout. This is acceptable to a 
> degree, but it appears that there are cases where a rogue query could execute 
> unmitigated. The attached testcase is an example of a query that will execute 
> and consume CPU/heap until an OutOfMemoryError is thrown.
> Example: The following query with 10s timeouts executed for ~7 minutes before 
> throwing an OOME.
> Aug 3, 2012 10:18:19 AM Executing query [limit=1,000 timeout1=10s 
> timeout2=10s]: SELECT * WHERE { ?a ?b ?c . ?c ?d ?e }
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.util.HashSet.(HashSet.java:86)
> at org.openjena.atlas.iterator.FilterUnique.(FilterUnique.java:26)
> at org.openjena.atlas.iterator.Iter.distinct(Iter.java:438)
> at 
> com.hp.hpl.jena.tdb.solver.StageMatchTuple.makeNextStage(StageMatchTuple.java:116)
> at 
> com.hp.hpl.jena.tdb.solver.StageMatchTuple.makeNextStage(StageMatchTuple.java:44)
> at 
> org.openjena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:49)
> at org.openjena.atlas.iterator.Iter$4.hasNext(Iter.java:295)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIterPlainWrapper.hasNextBinding(QueryIterPlainWrapper.java:54)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIterSlice.hasNextBinding(QueryIterSlice.java:76)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorWrapper.hasNextBinding(QueryIteratorWrapper.java:40)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorWrapper.hasNextBinding(QueryIteratorWrapper.java:40)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorWrapper.hasNextBinding(QueryIteratorWrapper.java:40)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
> at 
> com.hp.hpl.jena.sparql.engine.ResultSetStream.hasNext(ResultSetStream.java:72)
> at 
> com.hp.hpl.jena.sparql.resultset.ResultSetApply.apply(ResultSetApply.java:41)
> at com.hp.hpl.jena.sparql.resultset.XMLOutput.format(XMLOutput.java:52)
> at 
> com.hp.hpl.jena.query.ResultSetFormatter.outputAsXML(ResultSetFormatter.java:482)
> at 
> com.hp.hpl.jena.query.ResultSetFormatter.outputAsXML(ResultSetFormatter.java:460)
> at TestTimout.main(TestTimout.java:84)
> Aug 3, 2012 10:25:41 AM Finished

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-327) TDB Tx transaction lock to permit backups

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13464023#comment-13464023
 ] 

Simon Helsen commented on JENA-327:
---

ok, so if we write out n-quads in a read transaction and that takes a long 
time, TDB/Tx will keep serving queries and permitting updates, but it won't 
merge the journal during that time (to honor the transactional semantics). What 
about memory usage? I am trying to assess whether this approach would not cause 
other dangers to a running system. And of course, a restore in such cases will 
take as long as it takes to import the quads. I did some quick calculations and 
a store with 50 million quads would take 15 minutes on my machine, probably 
faster on decent server hardware.

> TDB Tx transaction lock to permit backups
> -
>
> Key: JENA-327
> URL: https://issues.apache.org/jira/browse/JENA-327
> Project: Apache Jena
>  Issue Type: Improvement
>  Components: TDB
>Affects Versions: TDB 0.9.4
>Reporter: Simon Helsen
>
> With large repositories, it is important to be able to create backups once in 
> a while. This is because recreating an rdf store with millions of triples can 
> be forbiddingly expensive. Moreover, it should be possible to take those 
> backups while still allowing read activity on the store as in many cases, a 
> complete shutdown is usually not possible. Before the introduction of tx, it 
> was relatively straightforward to provide the right locks on the client-side 
> to safely suspend any disk activity for a period of time enough to make a 
> backup of the index. 
> However, since tx, things have become slightly more complicated because TDB 
> Tx touches the disk at other times than when performing write/sync 
> activities. Right now, because of some understanding of how TDB Tx is 
> implemented, it is still possible for clients to avoid disk activities to 
> implement a backup process, but this dependency on TDB Tx implementation 
> details is not very good. Moreover, we anticipate that in the future, the 
> merging process from the journal into the main index may become entirely 
> asynchornous for performance reasons. The moment that happens, client have no 
> control anymore as to when the disk is being touched.
> For this reason, we are requesting the following feature: a "backup" lock (by 
> lack of a better name). Its semantics is that when the lock is taken, TDB Tx 
> guarantees that no disk activity takes place and if necessary pauses 
> activities. In other words, no write transaction should be able to complete 
> and read transactions will not attempt to merge the journal. The idea would 
> be that regular read activities can still continue. The API could be as 
> simple as something like this:
> try {
> dataset.begin(ReadWrite.BACKUP) ;
> 
> } finally {
> dataset.end()
> }
> As for the implementation, we suspect you currently have locks in place which 
> could be used to guarantee this behavior. E.g. could 
> txn.getBaseDataset().getLock().enterCriticalSection(Lock.WRITE) be sufficient?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-327) TDB Tx transaction lock to permit backups

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13463125#comment-13463125
 ] 

Simon Helsen commented on JENA-327:
---

"what counts as 'extremely large stores' in triple/quad count?" 

50million+, i.e. many gigabytes of disk space

"A READ action does not depend on internal characteristics so it is the most 
stable option"

ok, so let me make sure I get this right: you are saying that the best way to 
perform online backups is to perform a quad dump on the dataset inside a READ 
transaction and that this is guaranteed to remain safe over time? 

I am not following "There is no need to flush the journal - just back it up 
like everything else", Is this separately required when backing up from the 
regular dataset in the READ transaction? Or was this comment referring to a 
separate option. 



> TDB Tx transaction lock to permit backups
> -
>
> Key: JENA-327
> URL: https://issues.apache.org/jira/browse/JENA-327
> Project: Apache Jena
>  Issue Type: Improvement
>  Components: TDB
>Affects Versions: TDB 0.9.4
>Reporter: Simon Helsen
>
> With large repositories, it is important to be able to create backups once in 
> a while. This is because recreating an rdf store with millions of triples can 
> be forbiddingly expensive. Moreover, it should be possible to take those 
> backups while still allowing read activity on the store as in many cases, a 
> complete shutdown is usually not possible. Before the introduction of tx, it 
> was relatively straightforward to provide the right locks on the client-side 
> to safely suspend any disk activity for a period of time enough to make a 
> backup of the index. 
> However, since tx, things have become slightly more complicated because TDB 
> Tx touches the disk at other times than when performing write/sync 
> activities. Right now, because of some understanding of how TDB Tx is 
> implemented, it is still possible for clients to avoid disk activities to 
> implement a backup process, but this dependency on TDB Tx implementation 
> details is not very good. Moreover, we anticipate that in the future, the 
> merging process from the journal into the main index may become entirely 
> asynchornous for performance reasons. The moment that happens, client have no 
> control anymore as to when the disk is being touched.
> For this reason, we are requesting the following feature: a "backup" lock (by 
> lack of a better name). Its semantics is that when the lock is taken, TDB Tx 
> guarantees that no disk activity takes place and if necessary pauses 
> activities. In other words, no write transaction should be able to complete 
> and read transactions will not attempt to merge the journal. The idea would 
> be that regular read activities can still continue. The API could be as 
> simple as something like this:
> try {
> dataset.begin(ReadWrite.BACKUP) ;
> 
> } finally {
> dataset.end()
> }
> As for the implementation, we suspect you currently have locks in place which 
> could be used to guarantee this behavior. E.g. could 
> txn.getBaseDataset().getLock().enterCriticalSection(Lock.WRITE) be sufficient?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-327) TDB Tx transaction lock to permit backups

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13459725#comment-13459725
 ] 

Simon Helsen commented on JENA-327:
---

first of all, we currently do a backup by copying in the file system. Our 
initial impression was that this would be faster than any other approach 
especially given that the restore has to be fast as well. I have no numbers 
right now, but I think for now, we want to stick to this unless we have 
evidence the n-quad dump is fast enough on extremely large stores. 

So if I understand you well, you are saying that a WRITE transaction will 
guarantee that nothing changes on disk even if async writeback is introduced? 
That sounds good and would serve our purpose. Why would using the read 
mechanism be safer long-term? 

The 3rd option is not clear to me. Are you saying there is API (not internal 
interfaces) we can use to a) hold up everything and b) manually flush the 
journal?

> TDB Tx transaction lock to permit backups
> -
>
> Key: JENA-327
> URL: https://issues.apache.org/jira/browse/JENA-327
> Project: Apache Jena
>  Issue Type: Improvement
>  Components: TDB
>Affects Versions: TDB 0.9.4
>Reporter: Simon Helsen
>
> With large repositories, it is important to be able to create backups once in 
> a while. This is because recreating an rdf store with millions of triples can 
> be forbiddingly expensive. Moreover, it should be possible to take those 
> backups while still allowing read activity on the store as in many cases, a 
> complete shutdown is usually not possible. Before the introduction of tx, it 
> was relatively straightforward to provide the right locks on the client-side 
> to safely suspend any disk activity for a period of time enough to make a 
> backup of the index. 
> However, since tx, things have become slightly more complicated because TDB 
> Tx touches the disk at other times than when performing write/sync 
> activities. Right now, because of some understanding of how TDB Tx is 
> implemented, it is still possible for clients to avoid disk activities to 
> implement a backup process, but this dependency on TDB Tx implementation 
> details is not very good. Moreover, we anticipate that in the future, the 
> merging process from the journal into the main index may become entirely 
> asynchornous for performance reasons. The moment that happens, client have no 
> control anymore as to when the disk is being touched.
> For this reason, we are requesting the following feature: a "backup" lock (by 
> lack of a better name). Its semantics is that when the lock is taken, TDB Tx 
> guarantees that no disk activity takes place and if necessary pauses 
> activities. In other words, no write transaction should be able to complete 
> and read transactions will not attempt to merge the journal. The idea would 
> be that regular read activities can still continue. The API could be as 
> simple as something like this:
> try {
> dataset.begin(ReadWrite.BACKUP) ;
> 
> } finally {
> dataset.end()
> }
> As for the implementation, we suspect you currently have locks in place which 
> could be used to guarantee this behavior. E.g. could 
> txn.getBaseDataset().getLock().enterCriticalSection(Lock.WRITE) be sufficient?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-324) An exception thrown from commit() leaves the StoreConnection in an inconsistent state.

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13459682#comment-13459682
 ] 

Simon Helsen commented on JENA-324:
---

thanks for the response Andy. However, I am not sure how your comment addresses 
the concern. The problem we observe is that if something goes wrong during 
commit, it seems like a transaction gets in a stuck state which can only be 
recovered from when you call StoreConnection.expel(location, true), which does 
not seem official API

> An exception thrown from commit() leaves the StoreConnection in an 
> inconsistent state.
> --
>
> Key: JENA-324
> URL: https://issues.apache.org/jira/browse/JENA-324
> Project: Apache Jena
>  Issue Type: Bug
>  Components: TDB
>Affects Versions: TDB 0.9.1, TDB 0.9.2, TDB 0.9.3, TDB 0.9.4
>Reporter: Mark Buquor
>
> An exception thrown from commit() leaves the StoreConnection in an 
> inconsistent state.
> Example: After a commit() throws an IOException in prepare() due to 
> insufficient disk space, I see the following behavior:
> dataset.isInTransaction() == false
> dataset.abort() throws TDBTransactionException: "Transaction has already 
> committed or aborted"
> dataset.end() appears OK
> dataset.begin(ReadWrite.READ) appears OK
> dataset.begin(ReadWrite.WRITE) blocks on writersWaiting.acquire()
>  
> The debugger shows that the transaction is stuck in limbo: active, closed, 
> and unfinished.
> TransactionManager.activeWriters: 1
> TransactionManager.activeTransactions: 1
> [Transaction: 151 : Mode=WRITE : State=CLOSED : X:\RELM1\]
> changesPending=true
> outcome=UNFINISHED
> Short of a fix, it looks like the only option to clear the stuck transaction 
> is StoreConnection.expel(location, true), which has the comment "testing 
> only".
> There's also the indeterminacy of ending a transaction that, as far as I can 
> tell at the Dataset level, may or may not be committed ("Transaction has 
> already committed or aborted").
> Rough outline of a commit:
> state == TxnState.PREPARING
> -- commitPrepare()
> -- journal.write()
> -- journal.sync() // Commit point.
> state == TxnState.COMMITED
> -- noteTxnCommit()
> -- currentReaderView.set(null)
> -- writersWaiting.release()
> noteTxnCommit() is called after the commit and performs IO, so it appears 
> that if commit() throws an exception, it's not necessarily true that the 
> transaction is effectively aborted.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (JENA-327) TDB Tx transaction lock to permit backups

[Simon Helsen (JIRA)]

Simon Helsen created JENA-327:
-

 Summary: TDB Tx transaction lock to permit backups
 Key: JENA-327
 URL: https://issues.apache.org/jira/browse/JENA-327
 Project: Apache Jena
  Issue Type: Improvement
  Components: TDB
Affects Versions: TDB 0.9.4
Reporter: Simon Helsen


With large repositories, it is important to be able to create backups once in a 
while. This is because recreating an rdf store with millions of triples can be 
forbiddingly expensive. Moreover, it should be possible to take those backups 
while still allowing read activity on the store as in many cases, a complete 
shutdown is usually not possible. Before the introduction of tx, it was 
relatively straightforward to provide the right locks on the client-side to 
safely suspend any disk activity for a period of time enough to make a backup 
of the index. 

However, since tx, things have become slightly more complicated because TDB Tx 
touches the disk at other times than when performing write/sync activities. 
Right now, because of some understanding of how TDB Tx is implemented, it is 
still possible for clients to avoid disk activities to implement a backup 
process, but this dependency on TDB Tx implementation details is not very good. 
Moreover, we anticipate that in the future, the merging process from the 
journal into the main index may become entirely asynchornous for performance 
reasons. The moment that happens, client have no control anymore as to when the 
disk is being touched.

For this reason, we are requesting the following feature: a "backup" lock (by 
lack of a better name). Its semantics is that when the lock is taken, TDB Tx 
guarantees that no disk activity takes place and if necessary pauses 
activities. In other words, no write transaction should be able to complete and 
read transactions will not attempt to merge the journal. The idea would be that 
regular read activities can still continue. The API could be as simple as 
something like this:

try {
dataset.begin(ReadWrite.BACKUP) ;



} finally {
dataset.end()
}

As for the implementation, we suspect you currently have locks in place which 
could be used to guarantee this behavior. E.g. could 
txn.getBaseDataset().getLock().enterCriticalSection(Lock.WRITE) be sufficient?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-293) Allow the filter placement optimziation to push FILTER though BIND.

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13457809#comment-13457809
 ] 

Simon Helsen commented on JENA-293:
---

we ran into this. Not sure if this defect is on the radar

> Allow the filter placement optimziation to push FILTER though BIND.
> ---
>
> Key: JENA-293
> URL: https://issues.apache.org/jira/browse/JENA-293
> Project: Apache Jena
>  Issue Type: Improvement
>  Components: ARQ
>Reporter: Andy Seaborne
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-289) Respect query timeouts in TDB implementation

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13454932#comment-13454932
 ] 

Simon Helsen commented on JENA-289:
---

Discussed it offline with Mark. The real problem is that we may not be able to 
catch all dangerous queries during testing and when we hit this issue, it 
brings the entire system down. I agree that the severity should be raised

> Respect query timeouts in TDB implementation
> 
>
> Key: JENA-289
> URL: https://issues.apache.org/jira/browse/JENA-289
> Project: Apache Jena
>  Issue Type: Improvement
>  Components: TDB
>Affects Versions: TDB 0.9.1
>Reporter: Mark Buquor
>Priority: Critical
> Attachments: TestTimeout.java, TestTimeout.java
>
>
> In general use, we sometimes see queries throw QueryCancelledException 
> several seconds/minutes after the expected timeout. This is acceptable to a 
> degree, but it appears that there are cases where a rogue query could execute 
> unmitigated. The attached testcase is an example of a query that will execute 
> and consume CPU/heap until an OutOfMemoryError is thrown.
> Example: The following query with 10s timeouts executed for ~7 minutes before 
> throwing an OOME.
> Aug 3, 2012 10:18:19 AM Executing query [limit=1,000 timeout1=10s 
> timeout2=10s]: SELECT * WHERE { ?a ?b ?c . ?c ?d ?e }
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.util.HashSet.(HashSet.java:86)
> at org.openjena.atlas.iterator.FilterUnique.(FilterUnique.java:26)
> at org.openjena.atlas.iterator.Iter.distinct(Iter.java:438)
> at 
> com.hp.hpl.jena.tdb.solver.StageMatchTuple.makeNextStage(StageMatchTuple.java:116)
> at 
> com.hp.hpl.jena.tdb.solver.StageMatchTuple.makeNextStage(StageMatchTuple.java:44)
> at 
> org.openjena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:49)
> at org.openjena.atlas.iterator.Iter$4.hasNext(Iter.java:295)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIterPlainWrapper.hasNextBinding(QueryIterPlainWrapper.java:54)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIterSlice.hasNextBinding(QueryIterSlice.java:76)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorWrapper.hasNextBinding(QueryIteratorWrapper.java:40)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorWrapper.hasNextBinding(QueryIteratorWrapper.java:40)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorWrapper.hasNextBinding(QueryIteratorWrapper.java:40)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
> at 
> com.hp.hpl.jena.sparql.engine.ResultSetStream.hasNext(ResultSetStream.java:72)
> at 
> com.hp.hpl.jena.sparql.resultset.ResultSetApply.apply(ResultSetApply.java:41)
> at com.hp.hpl.jena.sparql.resultset.XMLOutput.format(XMLOutput.java:52)
> at 
> com.hp.hpl.jena.query.ResultSetFormatter.outputAsXML(ResultSetFormatter.java:482)
> at 
> com.hp.hpl.jena.query.ResultSetFormatter.outputAsXML(ResultSetFormatter.java:460)
> at TestTimout.main(TestTimout.java:84)
> Aug 3, 2012 10:25:41 AM Finished

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-244) Deadlock during SPARQL execution on an inference model

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13453983#comment-13453983
 ] 

Simon Helsen commented on JENA-244:
---

Have you tested this with a snapshot 2.7.4 build?

> Deadlock during SPARQL execution on an inference model
> --
>
> Key: JENA-244
> URL: https://issues.apache.org/jira/browse/JENA-244
> Project: Apache Jena
>  Issue Type: Bug
>  Components: Jena
>Reporter: Stephen Owens
> Attachments: JenaDeadLockTest.java, JenaDeadLockTest.java
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-289) Respect query timeouts in TDB implementation

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13452355#comment-13452355
 ] 

Simon Helsen commented on JENA-289:
---

Incidentally, we hit this issue again with another client. They just observed 
the OOME (They had 3,5Gb of data sitting in BindingTDB) and after deeper 
analysis found that a query was doing something really bad which caused the 
pathological explosion of the search space. Unlike the other client, they ran 
into the OME before realizing the query was never aborted.

but here as well, once can fix the query to bypass the problem

> Respect query timeouts in TDB implementation
> 
>
> Key: JENA-289
> URL: https://issues.apache.org/jira/browse/JENA-289
> Project: Apache Jena
>  Issue Type: Improvement
>  Components: TDB
>Affects Versions: TDB 0.9.1
>Reporter: Mark Buquor
> Attachments: TestTimeout.java, TestTimeout.java
>
>
> In general use, we sometimes see queries throw QueryCancelledException 
> several seconds/minutes after the expected timeout. This is acceptable to a 
> degree, but it appears that there are cases where a rogue query could execute 
> unmitigated. The attached testcase is an example of a query that will execute 
> and consume CPU/heap until an OutOfMemoryError is thrown.
> Example: The following query with 10s timeouts executed for ~7 minutes before 
> throwing an OOME.
> Aug 3, 2012 10:18:19 AM Executing query [limit=1,000 timeout1=10s 
> timeout2=10s]: SELECT * WHERE { ?a ?b ?c . ?c ?d ?e }
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.util.HashSet.(HashSet.java:86)
> at org.openjena.atlas.iterator.FilterUnique.(FilterUnique.java:26)
> at org.openjena.atlas.iterator.Iter.distinct(Iter.java:438)
> at 
> com.hp.hpl.jena.tdb.solver.StageMatchTuple.makeNextStage(StageMatchTuple.java:116)
> at 
> com.hp.hpl.jena.tdb.solver.StageMatchTuple.makeNextStage(StageMatchTuple.java:44)
> at 
> org.openjena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:49)
> at org.openjena.atlas.iterator.Iter$4.hasNext(Iter.java:295)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIterPlainWrapper.hasNextBinding(QueryIterPlainWrapper.java:54)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIterSlice.hasNextBinding(QueryIterSlice.java:76)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorWrapper.hasNextBinding(QueryIteratorWrapper.java:40)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorWrapper.hasNextBinding(QueryIteratorWrapper.java:40)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorWrapper.hasNextBinding(QueryIteratorWrapper.java:40)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
> at 
> com.hp.hpl.jena.sparql.engine.ResultSetStream.hasNext(ResultSetStream.java:72)
> at 
> com.hp.hpl.jena.sparql.resultset.ResultSetApply.apply(ResultSetApply.java:41)
> at com.hp.hpl.jena.sparql.resultset.XMLOutput.format(XMLOutput.java:52)
> at 
> com.hp.hpl.jena.query.ResultSetFormatter.outputAsXML(ResultSetFormatter.java:482)
> at 
> com.hp.hpl.jena.query.ResultSetFormatter.outputAsXML(ResultSetFormatter.java:460)
> at TestTimout.main(TestTimout.java:84)
> Aug 3, 2012 10:25:41 AM Finished

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-93) Permit cancelAllowDrain on a per query basis

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-93?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13452038#comment-13452038
 ] 

Simon Helsen commented on JENA-93:
--

I am ok with that. (Sorry for not mentioning this earlier - it really got of my 
radar this one was still open)

> Permit cancelAllowDrain on a per query basis
> 
>
> Key: JENA-93
> URL: https://issues.apache.org/jira/browse/JENA-93
> Project: Apache Jena
>  Issue Type: Improvement
>  Components: ARQ
>Reporter: Simon Helsen
>
> Currently, there is a global static variable 
> queryExecutionBase.cancelAllowDrain to permit partial results on cancelled 
> queries. 
> We would need this ability to be determinable on a per-query basis since we 
> have to mix both use-cases on our back-end and a the same time. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-289) Respect query timeouts in TDB implementation

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13452022#comment-13452022
 ] 

Simon Helsen commented on JENA-289:
---

We also observed this issue in a query whose execution plan led to a 
pathological explosion of the search space (on a very large repository). The 
query was not aborting after 10 hours or something. In practice however, these 
are non-issues since our clients, when writing such queries can be instructed 
to change the query to avoid pathological execution plans.

@Jena committers: although it would be great to get a fix for this issue, it is 
not critical to have it in a new service release (2.7.4). I am making this 
statement on behalf of the reporter of this defect as well (we clarified this 
in private communication). [~mark.buq...@us.ibm.com] FYI

> Respect query timeouts in TDB implementation
> 
>
> Key: JENA-289
> URL: https://issues.apache.org/jira/browse/JENA-289
> Project: Apache Jena
>  Issue Type: Improvement
>  Components: TDB
>Affects Versions: TDB 0.9.1
>Reporter: Mark Buquor
> Attachments: TestTimeout.java, TestTimeout.java
>
>
> In general use, we sometimes see queries throw QueryCancelledException 
> several seconds/minutes after the expected timeout. This is acceptable to a 
> degree, but it appears that there are cases where a rogue query could execute 
> unmitigated. The attached testcase is an example of a query that will execute 
> and consume CPU/heap until an OutOfMemoryError is thrown.
> Example: The following query with 10s timeouts executed for ~7 minutes before 
> throwing an OOME.
> Aug 3, 2012 10:18:19 AM Executing query [limit=1,000 timeout1=10s 
> timeout2=10s]: SELECT * WHERE { ?a ?b ?c . ?c ?d ?e }
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.util.HashSet.(HashSet.java:86)
> at org.openjena.atlas.iterator.FilterUnique.(FilterUnique.java:26)
> at org.openjena.atlas.iterator.Iter.distinct(Iter.java:438)
> at 
> com.hp.hpl.jena.tdb.solver.StageMatchTuple.makeNextStage(StageMatchTuple.java:116)
> at 
> com.hp.hpl.jena.tdb.solver.StageMatchTuple.makeNextStage(StageMatchTuple.java:44)
> at 
> org.openjena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:49)
> at org.openjena.atlas.iterator.Iter$4.hasNext(Iter.java:295)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIterPlainWrapper.hasNextBinding(QueryIterPlainWrapper.java:54)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIterSlice.hasNextBinding(QueryIterSlice.java:76)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorWrapper.hasNextBinding(QueryIteratorWrapper.java:40)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorWrapper.hasNextBinding(QueryIteratorWrapper.java:40)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorWrapper.hasNextBinding(QueryIteratorWrapper.java:40)
> at 
> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
> at 
> com.hp.hpl.jena.sparql.engine.ResultSetStream.hasNext(ResultSetStream.java:72)
> at 
> com.hp.hpl.jena.sparql.resultset.ResultSetApply.apply(ResultSetApply.java:41)
> at com.hp.hpl.jena.sparql.resultset.XMLOutput.format(XMLOutput.java:52)
> at 
> com.hp.hpl.jena.query.ResultSetFormatter.outputAsXML(ResultSetFormatter.java:482)
> at 
> com.hp.hpl.jena.query.ResultSetFormatter.outputAsXML(ResultSetFormatter.java:460)
> at TestTimout.main(TestTimout.java:84)
> Aug 3, 2012 10:25:41 AM Finished

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-294) TransformFilterEquality does not handle starting OPTIONAL well

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13451979#comment-13451979
 ] 

Simon Helsen commented on JENA-294:
---

although I would like this optimization problem be fixed at some point, we 
managed to get our client to change their query generator to bypass the 
problem. We are not aware of anyone else relying on the pattern, so for us the 
prioity is much lower now. It should certainly not inhibit a new release

> TransformFilterEquality does not handle starting OPTIONAL well
> --
>
> Key: JENA-294
> URL: https://issues.apache.org/jira/browse/JENA-294
> Project: Apache Jena
>  Issue Type: Improvement
>  Components: ARQ
>Affects Versions: Jena 2.7.4
>Reporter: Simon Helsen
>Assignee: Andy Seaborne
> Fix For: ARQ 2.9.4
>
> Attachments: filter-equality-13.rq, filter-equality-13.srj, 
> jena-294-afs-1.patch, manifestPatchForOptFilterEquality.txt, patch.txt, 
> patch.txt, test-filter-equality.zip
>
>
> There was one other case where our tests were stuck on a very slow query 
> execution because transformFilterEquality failed to optimize. The problem is 
> that the optimizer gives up whenever the WHERE clause starts with an OPTIONAL 
> clause. The reason is that the generated algebraic formula starts with a 
> TableUnit and this is not handled correctly. 
> I have attached a patch which fixes the problem

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-93) Permit cancelAllowDrain on a per query basis

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-93?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13451976#comment-13451976
 ] 

Simon Helsen commented on JENA-93:
--

This feature request is no longer relevant to us since we were able to convince 
our clients that working partial results causes complications and should be 
worked around. So we removed support for the concept entirely.

So, as far as I am concerned, this can be closed unless you feel it is a useful 
feature in general, other may benefit from

> Permit cancelAllowDrain on a per query basis
> 
>
> Key: JENA-93
> URL: https://issues.apache.org/jira/browse/JENA-93
> Project: Apache Jena
>  Issue Type: Improvement
>  Components: ARQ
>Reporter: Simon Helsen
>
> Currently, there is a global static variable 
> queryExecutionBase.cancelAllowDrain to permit partial results on cancelled 
> queries. 
> We would need this ability to be determinable on a per-query basis since we 
> have to mix both use-cases on our back-end and a the same time. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-311) Quad: object cannot be null

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13449760#comment-13449760
 ] 

Simon Helsen commented on JENA-311:
---

I am able to confirm that the issue is resolved

> Quad: object cannot be null
> ---
>
> Key: JENA-311
> URL: https://issues.apache.org/jira/browse/JENA-311
> Project: Apache Jena
>  Issue Type: Bug
>  Components: TDB
>Affects Versions: TDB 0.9.4
> Environment: windows 7/Oracle 6 JVM
>Reporter: Simon Helsen
>Assignee: Andy Seaborne
> Fix For: TDB 0.9.4
>
> Attachments: QuadsObjectIsNullTest.zip, T_QuadsObjectIsNull.java
>
>
> I have attached a test case. If you first execute it and then execute it 
> again you will hit
> Exception in thread "main" java.lang.UnsupportedOperationException: Quad: 
> object cannot be null
>   at com.hp.hpl.jena.sparql.core.Quad.(Quad.java:62)
>   at com.hp.hpl.jena.tdb.lib.TupleLib.quad(TupleLib.java:162)
>   at com.hp.hpl.jena.tdb.lib.TupleLib.quad(TupleLib.java:153)
>   at com.hp.hpl.jena.tdb.lib.TupleLib.access$1(TupleLib.java:149)
>   at com.hp.hpl.jena.tdb.lib.TupleLib$4.convert(TupleLib.java:87)
>   at com.hp.hpl.jena.tdb.lib.TupleLib$4.convert(TupleLib.java:1)
>   at org.openjena.atlas.iterator.Iter$4.next(Iter.java:301)
>   at org.openjena.atlas.iterator.IteratorCons.next(IteratorCons.java:97)
>   at org.openjena.atlas.iterator.Iter.sendToSink(Iter.java:572)
>   at org.openjena.riot.out.NQuadsWriter.write(NQuadsWriter.java:51)
>   at org.openjena.riot.out.NQuadsWriter.write(NQuadsWriter.java:38)
>   at org.openjena.riot.RiotWriter.writeNQuads(RiotWriter.java:41)
>   at 
> com.ibm.jena.test.QuadsObjectIsNullTest.main(QuadsObjectIsNullTest.java:147)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-311) Quad: object cannot be null

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13446417#comment-13446417
 ] 

Simon Helsen commented on JENA-311:
---

I forgot to mention that this problem did not exist in 0.9.1, but we started 
seeing it since 0.9.4

> Quad: object cannot be null
> ---
>
> Key: JENA-311
> URL: https://issues.apache.org/jira/browse/JENA-311
> Project: Apache Jena
>  Issue Type: Bug
>  Components: TDB
>Affects Versions: TDB 0.9.4
> Environment: windows 7/Oracle 6 JVM
>Reporter: Simon Helsen
> Attachments: QuadsObjectIsNullTest.zip
>
>
> I have attached a test case. If you first execute it and then execute it 
> again you will hit
> Exception in thread "main" java.lang.UnsupportedOperationException: Quad: 
> object cannot be null
>   at com.hp.hpl.jena.sparql.core.Quad.(Quad.java:62)
>   at com.hp.hpl.jena.tdb.lib.TupleLib.quad(TupleLib.java:162)
>   at com.hp.hpl.jena.tdb.lib.TupleLib.quad(TupleLib.java:153)
>   at com.hp.hpl.jena.tdb.lib.TupleLib.access$1(TupleLib.java:149)
>   at com.hp.hpl.jena.tdb.lib.TupleLib$4.convert(TupleLib.java:87)
>   at com.hp.hpl.jena.tdb.lib.TupleLib$4.convert(TupleLib.java:1)
>   at org.openjena.atlas.iterator.Iter$4.next(Iter.java:301)
>   at org.openjena.atlas.iterator.IteratorCons.next(IteratorCons.java:97)
>   at org.openjena.atlas.iterator.Iter.sendToSink(Iter.java:572)
>   at org.openjena.riot.out.NQuadsWriter.write(NQuadsWriter.java:51)
>   at org.openjena.riot.out.NQuadsWriter.write(NQuadsWriter.java:38)
>   at org.openjena.riot.RiotWriter.writeNQuads(RiotWriter.java:41)
>   at 
> com.ibm.jena.test.QuadsObjectIsNullTest.main(QuadsObjectIsNullTest.java:147)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (JENA-311) Quad: object cannot be null

[Simon Helsen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/JENA-311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Simon Helsen updated JENA-311:
--

Attachment: (was: QuadsObjectIsNullTest.zip)

> Quad: object cannot be null
> ---
>
> Key: JENA-311
> URL: https://issues.apache.org/jira/browse/JENA-311
> Project: Apache Jena
>  Issue Type: Bug
>  Components: TDB
>Affects Versions: TDB 0.9.4
> Environment: windows 7/Oracle 6 JVM
>Reporter: Simon Helsen
>
> I have attached a test case. If you first execute it and then execute it 
> again you will hit
> Exception in thread "main" java.lang.UnsupportedOperationException: Quad: 
> object cannot be null
>   at com.hp.hpl.jena.sparql.core.Quad.(Quad.java:62)
>   at com.hp.hpl.jena.tdb.lib.TupleLib.quad(TupleLib.java:162)
>   at com.hp.hpl.jena.tdb.lib.TupleLib.quad(TupleLib.java:153)
>   at com.hp.hpl.jena.tdb.lib.TupleLib.access$1(TupleLib.java:149)
>   at com.hp.hpl.jena.tdb.lib.TupleLib$4.convert(TupleLib.java:87)
>   at com.hp.hpl.jena.tdb.lib.TupleLib$4.convert(TupleLib.java:1)
>   at org.openjena.atlas.iterator.Iter$4.next(Iter.java:301)
>   at org.openjena.atlas.iterator.IteratorCons.next(IteratorCons.java:97)
>   at org.openjena.atlas.iterator.Iter.sendToSink(Iter.java:572)
>   at org.openjena.riot.out.NQuadsWriter.write(NQuadsWriter.java:51)
>   at org.openjena.riot.out.NQuadsWriter.write(NQuadsWriter.java:38)
>   at org.openjena.riot.RiotWriter.writeNQuads(RiotWriter.java:41)
>   at 
> com.ibm.jena.test.QuadsObjectIsNullTest.main(QuadsObjectIsNullTest.java:147)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (JENA-311) Quad: object cannot be null

[Simon Helsen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/JENA-311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Simon Helsen updated JENA-311:
--

Attachment: QuadsObjectIsNullTest.zip

> Quad: object cannot be null
> ---
>
> Key: JENA-311
> URL: https://issues.apache.org/jira/browse/JENA-311
> Project: Apache Jena
>  Issue Type: Bug
>  Components: TDB
>Affects Versions: TDB 0.9.4
> Environment: windows 7/Oracle 6 JVM
>Reporter: Simon Helsen
> Attachments: QuadsObjectIsNullTest.zip
>
>
> I have attached a test case. If you first execute it and then execute it 
> again you will hit
> Exception in thread "main" java.lang.UnsupportedOperationException: Quad: 
> object cannot be null
>   at com.hp.hpl.jena.sparql.core.Quad.(Quad.java:62)
>   at com.hp.hpl.jena.tdb.lib.TupleLib.quad(TupleLib.java:162)
>   at com.hp.hpl.jena.tdb.lib.TupleLib.quad(TupleLib.java:153)
>   at com.hp.hpl.jena.tdb.lib.TupleLib.access$1(TupleLib.java:149)
>   at com.hp.hpl.jena.tdb.lib.TupleLib$4.convert(TupleLib.java:87)
>   at com.hp.hpl.jena.tdb.lib.TupleLib$4.convert(TupleLib.java:1)
>   at org.openjena.atlas.iterator.Iter$4.next(Iter.java:301)
>   at org.openjena.atlas.iterator.IteratorCons.next(IteratorCons.java:97)
>   at org.openjena.atlas.iterator.Iter.sendToSink(Iter.java:572)
>   at org.openjena.riot.out.NQuadsWriter.write(NQuadsWriter.java:51)
>   at org.openjena.riot.out.NQuadsWriter.write(NQuadsWriter.java:38)
>   at org.openjena.riot.RiotWriter.writeNQuads(RiotWriter.java:41)
>   at 
> com.ibm.jena.test.QuadsObjectIsNullTest.main(QuadsObjectIsNullTest.java:147)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (JENA-311) Quad: object cannot be null

[Simon Helsen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/JENA-311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Simon Helsen updated JENA-311:
--

Attachment: QuadsObjectIsNullTest.zip

Note that in the test I write the resource 3 times. This is required. I am not 
hitting the problem when I only write 2 or 1 resources. 

> Quad: object cannot be null
> ---
>
> Key: JENA-311
> URL: https://issues.apache.org/jira/browse/JENA-311
> Project: Apache Jena
>  Issue Type: Bug
>  Components: TDB
>Affects Versions: TDB 0.9.4
> Environment: windows 7/Oracle 6 JVM
>Reporter: Simon Helsen
> Attachments: QuadsObjectIsNullTest.zip
>
>
> I have attached a test case. If you first execute it and then execute it 
> again you will hit
> Exception in thread "main" java.lang.UnsupportedOperationException: Quad: 
> object cannot be null
>   at com.hp.hpl.jena.sparql.core.Quad.(Quad.java:62)
>   at com.hp.hpl.jena.tdb.lib.TupleLib.quad(TupleLib.java:162)
>   at com.hp.hpl.jena.tdb.lib.TupleLib.quad(TupleLib.java:153)
>   at com.hp.hpl.jena.tdb.lib.TupleLib.access$1(TupleLib.java:149)
>   at com.hp.hpl.jena.tdb.lib.TupleLib$4.convert(TupleLib.java:87)
>   at com.hp.hpl.jena.tdb.lib.TupleLib$4.convert(TupleLib.java:1)
>   at org.openjena.atlas.iterator.Iter$4.next(Iter.java:301)
>   at org.openjena.atlas.iterator.IteratorCons.next(IteratorCons.java:97)
>   at org.openjena.atlas.iterator.Iter.sendToSink(Iter.java:572)
>   at org.openjena.riot.out.NQuadsWriter.write(NQuadsWriter.java:51)
>   at org.openjena.riot.out.NQuadsWriter.write(NQuadsWriter.java:38)
>   at org.openjena.riot.RiotWriter.writeNQuads(RiotWriter.java:41)
>   at 
> com.ibm.jena.test.QuadsObjectIsNullTest.main(QuadsObjectIsNullTest.java:147)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (JENA-311) Quad: object cannot be null

[Simon Helsen (JIRA)]

Simon Helsen created JENA-311:
-

 Summary: Quad: object cannot be null
 Key: JENA-311
 URL: https://issues.apache.org/jira/browse/JENA-311
 Project: Apache Jena
  Issue Type: Bug
  Components: TDB
Affects Versions: TDB 0.9.4
 Environment: windows 7/Oracle 6 JVM
Reporter: Simon Helsen


I have attached a test case. If you first execute it and then execute it again 
you will hit

Exception in thread "main" java.lang.UnsupportedOperationException: Quad: 
object cannot be null
at com.hp.hpl.jena.sparql.core.Quad.(Quad.java:62)
at com.hp.hpl.jena.tdb.lib.TupleLib.quad(TupleLib.java:162)
at com.hp.hpl.jena.tdb.lib.TupleLib.quad(TupleLib.java:153)
at com.hp.hpl.jena.tdb.lib.TupleLib.access$1(TupleLib.java:149)
at com.hp.hpl.jena.tdb.lib.TupleLib$4.convert(TupleLib.java:87)
at com.hp.hpl.jena.tdb.lib.TupleLib$4.convert(TupleLib.java:1)
at org.openjena.atlas.iterator.Iter$4.next(Iter.java:301)
at org.openjena.atlas.iterator.IteratorCons.next(IteratorCons.java:97)
at org.openjena.atlas.iterator.Iter.sendToSink(Iter.java:572)
at org.openjena.riot.out.NQuadsWriter.write(NQuadsWriter.java:51)
at org.openjena.riot.out.NQuadsWriter.write(NQuadsWriter.java:38)
at org.openjena.riot.RiotWriter.writeNQuads(RiotWriter.java:41)
at 
com.ibm.jena.test.QuadsObjectIsNullTest.main(QuadsObjectIsNullTest.java:147)


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-311) Quad: object cannot be null

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13446410#comment-13446410
 ] 

Simon Helsen commented on JENA-311:
---

The first run shows the following output:

17:45:07,758 [main] DEBUG com.hp.hpl.jena.tdb.transaction.TransactionManager  - 
Txn[1]/W: begin$
17:45:07,790 [main] DEBUG com.hp.hpl.jena.tdb.transaction.TransactionManager  - 
Txn[1]/W: begin
17:45:07,927 [main] DEBUG com.hp.hpl.jena.tdb.transaction.TransactionManager  - 
Txn[1]/W: commit
17:45:07,927 [main] DEBUG com.hp.hpl.jena.tdb.transaction.TransactionManager  - 
Txn[1]/W: Add to pending queue
17:45:07,927 [main] DEBUG com.hp.hpl.jena.tdb.transaction.TransactionManager  - 
Txn[2]/W: begin$
17:45:07,929 [main] DEBUG com.hp.hpl.jena.tdb.transaction.TransactionManager  - 
Txn[2]/W: begin
17:45:07,933 [main] DEBUG com.hp.hpl.jena.tdb.transaction.TransactionManager  - 
Txn[2]/W: commit
17:45:07,933 [main] DEBUG com.hp.hpl.jena.tdb.transaction.TransactionManager  - 
Txn[2]/W: Add to pending queue
17:45:07,934 [main] DEBUG com.hp.hpl.jena.tdb.transaction.TransactionManager  - 
Txn[3]/W: begin$
17:45:07,935 [main] DEBUG com.hp.hpl.jena.tdb.transaction.TransactionManager  - 
Txn[3]/W: begin
17:45:07,939 [main] DEBUG com.hp.hpl.jena.tdb.transaction.TransactionManager  - 
Txn[3]/W: commit
17:45:07,939 [main] DEBUG com.hp.hpl.jena.tdb.transaction.TransactionManager  - 
Txn[3]/W: Add to pending queue
17:45:07,940 [main] DEBUG com.hp.hpl.jena.tdb.transaction.TransactionManager  - 
Txn[4]/W: begin$
17:45:07,941 [main] DEBUG com.hp.hpl.jena.tdb.transaction.TransactionManager  - 
Txn[4]/W: begin
  "foo1" 
 .
  "foo1" 
 .
  "foo3" 
 .
17:45:07,951 [main] DEBUG com.hp.hpl.jena.tdb.transaction.TransactionManager  - 
Txn[4]/W: commit
17:45:07,951 [main] DEBUG com.hp.hpl.jena.tdb.transaction.TransactionManager  - 
Txn[4]/W: Add to pending queue

The second run then shows the following:

17:45:29,534 [main] DEBUG com.hp.hpl.jena.tdb.transaction.TransactionManager  - 
Txn[1]/W: begin$
17:45:29,562 [main] DEBUG com.hp.hpl.jena.tdb.transaction.TransactionManager  - 
Txn[1]/W: begin
17:45:29,697 [main] DEBUG com.hp.hpl.jena.tdb.transaction.TransactionManager  - 
Txn[1]/W: commit
17:45:29,697 [main] DEBUG com.hp.hpl.jena.tdb.transaction.TransactionManager  - 
Txn[1]/W: Add to pending queue
17:45:29,698 [main] DEBUG com.hp.hpl.jena.tdb.transaction.TransactionManager  - 
Txn[2]/W: begin$
17:45:29,699 [main] DEBUG com.hp.hpl.jena.tdb.transaction.TransactionManager  - 
Txn[2]/W: begin
17:45:29,702 [main] DEBUG com.hp.hpl.jena.tdb.transaction.TransactionManager  - 
Txn[2]/W: commit
17:45:29,702 [main] DEBUG com.hp.hpl.jena.tdb.transaction.TransactionManager  - 
Txn[2]/W: Add to pending queue
17:45:29,702 [main] DEBUG com.hp.hpl.jena.tdb.transaction.TransactionManager  - 
Txn[3]/W: begin$
17:45:29,704 [main] DEBUG com.hp.hpl.jena.tdb.transaction.TransactionManager  - 
Txn[3]/W: begin
17:45:29,706 [main] DEBUG com.hp.hpl.jena.tdb.transaction.TransactionManager  - 
Txn[3]/W: commit
17:45:29,706 [main] DEBUG com.hp.hpl.jena.tdb.transaction.TransactionManager  - 
Txn[3]/W: Add to pending queue
17:45:29,706 [main] DEBUG com.hp.hpl.jena.tdb.transaction.TransactionManager  - 
Txn[4]/W: begin$
17:45:29,708 [main] DEBUG com.hp.hpl.jena.tdb.transaction.TransactionManager  - 
Txn[4]/W: begin
Exception in thread "main" java.lang.UnsupportedOperationException: Quad: 
object cannot be null
at com.hp.hpl.jena.sparql.core.Quad.(Quad.java:62)
at com.hp.hpl.jena.tdb.lib.TupleLib.quad(TupleLib.java:162)
at com.hp.hpl.jena.tdb.lib.TupleLib.quad(TupleLib.java:153)
at com.hp.hpl.jena.tdb.lib.TupleLib.access$1(TupleLib.java:149)
at com.hp.hpl.jena.tdb.lib.TupleLib$4.convert(TupleLib.java:87)
at com.hp.hpl.jena.tdb.lib.TupleLib$4.convert(TupleLib.java:1)
at org.openjena.atlas.iterator.Iter$4.next(Iter.java:301)
at org.openjena.atlas.iterator.IteratorCons.next(IteratorCons.java:97)
at org.openjena.atlas.iterator.Iter.sendToSink(Iter.java:572)
at org.openjena.riot.out.NQuadsWriter.write(NQuadsWriter.java:51)
at org.openjena.riot.out.NQuadsWriter.write(NQuadsWriter.java:38)
at org.openjena.riot.RiotWriter.writeNQuads(RiotWriter.java:41)
at 
com.ibm.jena.test.QuadsObjectIsNullTest.main(QuadsObjectIsNullTest.java:147)


> Quad: object cannot be null
> ---
>
> Key: JENA-311
> URL: https://issues.apache.org/jira/browse/JENA-311
> Project: Apache Jena
>  Issue Type: Bug
>  Components: TDB
>Affects Versions: TDB 0.9.4
> 

[jira] [Commented] (JENA-308) Index corruption after killing process

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13446374#comment-13446374
 ] 

Simon Helsen commented on JENA-308:
---

looks good for now. I would need to see what happens in our system verification 
tests once we adopt a successor release of course

> Index corruption after killing process
> --
>
> Key: JENA-308
> URL: https://issues.apache.org/jira/browse/JENA-308
> Project: Apache Jena
>  Issue Type: Bug
>  Components: TDB
>Affects Versions: TDB 0.9.1, TDB 0.9.4
> Environment: Windows 2008/7 (64 bit) both on IBM and SUN JVM 6. 
> Direct mode
>Reporter: Simon Helsen
>Assignee: Andy Seaborne
>Priority: Critical
> Fix For: TDB 0.9.4
>
> Attachments: journalRecoveryTest.zip
>
>
> We are faced with a series of possible exceptions which may or may not occur 
> after an active tdb store is killed. The most common exception has the form:
> Exception in thread "Thread-3" 
> com.hp.hpl.jena.tdb.transaction.TDBTransactionException: Feaild to read block 
> checksum.
>   at com.hp.hpl.jena.tdb.transaction.Journal._read(Journal.java:238)
>   at 
> com.hp.hpl.jena.tdb.transaction.Journal._readJournal(Journal.java:197)
>   at com.hp.hpl.jena.tdb.transaction.Journal.access$1(Journal.java:192)
>   at 
> com.hp.hpl.jena.tdb.transaction.Journal$IteratorEntries.moveToNext(Journal.java:267)
>   at 
> com.hp.hpl.jena.tdb.transaction.Journal$IteratorEntries.moveToNext(Journal.java:1)
>   at 
> org.openjena.atlas.iterator.IteratorSlotted.hasNext(IteratorSlotted.java:67)
>   at 
> com.hp.hpl.jena.tdb.transaction.JournalControl.scanForCommit(JournalControl.java:148)
>   at 
> com.hp.hpl.jena.tdb.transaction.JournalControl.recoverFromJournal(JournalControl.java:127)
>   at 
> com.hp.hpl.jena.tdb.StoreConnection._makeAndCache(StoreConnection.java:234)
>   at com.hp.hpl.jena.tdb.StoreConnection.make(StoreConnection.java:214)
>   at 
> com.hp.hpl.jena.tdb.transaction.DatasetGraphTransaction.(DatasetGraphTransaction.java:76)
>   at com.hp.hpl.jena.tdb.sys.TDBMaker._create(TDBMaker.java:57)
>   at 
> com.hp.hpl.jena.tdb.sys.TDBMaker.createDatasetGraphTransaction(TDBMaker.java:45)
>   at 
> com.hp.hpl.jena.tdb.TDBFactory._createDatasetGraph(TDBFactory.java:104)
>   at com.hp.hpl.jena.tdb.TDBFactory.createDatasetGraph(TDBFactory.java:73)
>   at com.hp.hpl.jena.tdb.TDBFactory.createDataset(TDBFactory.java:52)
> Other exceptions are possible (see comments below). I have been able to 
> produce a standalone java program/test case which simulates a process kill 
> and is able to produce the exception after a non-deterministic time. It runs 
> an iteration until it hits one of the possible problems. Sometimes, an 
> exception hits after just 2 iterations, other times, it can take 40 runs or 
> more before hitting a problem. Because I don't know what the cause is here, I 
> have not been able to make the test case more specific

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-308) Index corruption after killing process

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13445313#comment-13445313
 ] 

Simon Helsen commented on JENA-308:
---

Thanks Andy, I will certainly test it out. I rarely had to run it more than 40 
iterations to hit any of the indicated exceptions, but yes, it is not ideal 
test code. 

> Index corruption after killing process
> --
>
> Key: JENA-308
> URL: https://issues.apache.org/jira/browse/JENA-308
> Project: Apache Jena
>  Issue Type: Bug
>  Components: TDB
>Affects Versions: TDB 0.9.1, TDB 0.9.4
> Environment: Windows 2008/7 (64 bit) both on IBM and SUN JVM 6. 
> Direct mode
>Reporter: Simon Helsen
>Assignee: Andy Seaborne
>Priority: Critical
> Fix For: TDB 0.9.4
>
> Attachments: journalRecoveryTest.zip
>
>
> We are faced with a series of possible exceptions which may or may not occur 
> after an active tdb store is killed. The most common exception has the form:
> Exception in thread "Thread-3" 
> com.hp.hpl.jena.tdb.transaction.TDBTransactionException: Feaild to read block 
> checksum.
>   at com.hp.hpl.jena.tdb.transaction.Journal._read(Journal.java:238)
>   at 
> com.hp.hpl.jena.tdb.transaction.Journal._readJournal(Journal.java:197)
>   at com.hp.hpl.jena.tdb.transaction.Journal.access$1(Journal.java:192)
>   at 
> com.hp.hpl.jena.tdb.transaction.Journal$IteratorEntries.moveToNext(Journal.java:267)
>   at 
> com.hp.hpl.jena.tdb.transaction.Journal$IteratorEntries.moveToNext(Journal.java:1)
>   at 
> org.openjena.atlas.iterator.IteratorSlotted.hasNext(IteratorSlotted.java:67)
>   at 
> com.hp.hpl.jena.tdb.transaction.JournalControl.scanForCommit(JournalControl.java:148)
>   at 
> com.hp.hpl.jena.tdb.transaction.JournalControl.recoverFromJournal(JournalControl.java:127)
>   at 
> com.hp.hpl.jena.tdb.StoreConnection._makeAndCache(StoreConnection.java:234)
>   at com.hp.hpl.jena.tdb.StoreConnection.make(StoreConnection.java:214)
>   at 
> com.hp.hpl.jena.tdb.transaction.DatasetGraphTransaction.(DatasetGraphTransaction.java:76)
>   at com.hp.hpl.jena.tdb.sys.TDBMaker._create(TDBMaker.java:57)
>   at 
> com.hp.hpl.jena.tdb.sys.TDBMaker.createDatasetGraphTransaction(TDBMaker.java:45)
>   at 
> com.hp.hpl.jena.tdb.TDBFactory._createDatasetGraph(TDBFactory.java:104)
>   at com.hp.hpl.jena.tdb.TDBFactory.createDatasetGraph(TDBFactory.java:73)
>   at com.hp.hpl.jena.tdb.TDBFactory.createDataset(TDBFactory.java:52)
> Other exceptions are possible (see comments below). I have been able to 
> produce a standalone java program/test case which simulates a process kill 
> and is able to produce the exception after a non-deterministic time. It runs 
> an iteration until it hits one of the possible problems. Sometimes, an 
> exception hits after just 2 iterations, other times, it can take 40 runs or 
> more before hitting a problem. Because I don't know what the cause is here, I 
> have not been able to make the test case more specific

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-301) RecordRangeIterator: records not strictly increasing

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13445017#comment-13445017
 ] 

Simon Helsen commented on JENA-301:
---

Andy, we recently ran into the following issue and I wonder if it could have 
the same origin (this was on 0.9.1). Any ideas?

Caused by: com.hp.hpl.jena.tdb.base.block.BlockException: No such block
at 
com.hp.hpl.jena.tdb.base.recordbuffer.RecordRangeIterator.iterator(RecordRangeIterator.java:39)
at 
com.hp.hpl.jena.tdb.index.bplustree.BPlusTree.iterator(BPlusTree.java:383)
at 
com.hp.hpl.jena.tdb.index.bplustree.BPlusTree.iterator(BPlusTree.java:366)
at 
com.hp.hpl.jena.tdb.index.TupleIndexRecord.findWorker(TupleIndexRecord.java:164)
at 
com.hp.hpl.jena.tdb.index.TupleIndexRecord.findOrScan(TupleIndexRecord.java:84)
at 
com.hp.hpl.jena.tdb.index.TupleIndexRecord.performFind(TupleIndexRecord.java:78)
at com.hp.hpl.jena.tdb.index.TupleIndexBase.find(TupleIndexBase.java:91)
at com.hp.hpl.jena.tdb.index.TupleTable.find(TupleTable.java:172)
at 
com.hp.hpl.jena.tdb.nodetable.NodeTupleTableConcrete.find(NodeTupleTableConcrete.java:169)
at 
com.hp.hpl.jena.tdb.solver.StageMatchTuple.makeNextStage(StageMatchTuple.java:101)
at 
com.hp.hpl.jena.tdb.solver.StageMatchTuple.makeNextStage(StageMatchTuple.java:44)
at 
org.openjena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:49)
at org.openjena.atlas.iterator.Iter$4.hasNext(Iter.java:295)
at 
com.hp.hpl.jena.sparql.engine.iterator.QueryIterPlainWrapper.hasNextBinding(QueryIterPlainWrapper.java:54)
at 
com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
at 
com.hp.hpl.jena.sparql.engine.iterator.QueryIterDefaulting.hasNextBinding(QueryIterDefaulting.java:54)
at 
com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
at 
com.hp.hpl.jena.sparql.engine.iterator.QueryIterRepeatApply.hasNextBinding(QueryIterRepeatApply.java:79)
at 
com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
at 
com.hp.hpl.jena.sparql.engine.iterator.QueryIterProcessBinding.hasNextBinding(QueryIterProcessBinding.java:60)
at 
com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
at 
com.hp.hpl.jena.sparql.engine.iterator.QueryIterConvert.hasNextBinding(QueryIterConvert.java:65)
at 
com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
at 
com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorWrapper.hasNextBinding(QueryIteratorWrapper.java:40)
at 
com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
at 
com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorWrapper.hasNextBinding(QueryIteratorWrapper.java:40)
at 
com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:108)
at 
com.hp.hpl.jena.sparql.engine.ResultSetStream.hasNext(ResultSetStream.java:72)
at 
com.ibm.team.jfs.rdf.internal.jena.InternalResultSet.(InternalResultSet.java:26)
at 
com.ibm.team.jfs.rdf.internal.jena.tdb.JenaTxTdbProvider.performSelect(JenaTxTdbProvider.java:1406)
at 
com.ibm.team.jfs.rdf.internal.jena.tdb.JenaTxTdbProvider$15.run(JenaTxTdbProvider.java:1537)
at 
com.ibm.team.jfs.rdf.internal.jena.tdb.JenaTxTdbProvider$15.run(JenaTxTdbProvider.java:1)
at 
com.ibm.team.jfs.rdf.internal.jena.tdb.JenaTxTdbProvider.storeOperation(JenaTxTdbProvider.java:180)

> RecordRangeIterator: records not strictly increasing
> 
>
> Key: JENA-301
> URL: https://issues.apache.org/jira/browse/JENA-301
> Project: Apache Jena
>  Issue Type: Bug
>  Components: TDB
>Affects Versions: TDB 0.9.4
> Environment: windows 64 bit, oracle JRE 6
>Reporter: Simon Helsen
>Assignee: Andy Seaborne
> Attachments: PerformanceRegressionTest.java
>
>
> When I tried to execute the provided test case for JENA-256, I ran into a TDB 
> Storage Exception. The scenario is like this
> 1) run the test application on a clean non-existing index. All should go 
> well, i.e. not exceptions, tests finish fine
> 2) then run the test application again on the same index. You'll end up with 
> the following stack trace at write operation 98:
> xception in thread "main" java.lang.RuntimeException: 
> com.hp.hpl.jena.tdb.base.StorageException: RecordRangeIterator: records not 
> strictly increasing: 
> 20ba002401490ad3 // 
> 08bc00240149027

[jira] [Commented] (JENA-308) Index corruption after killing process

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13444984#comment-13444984
 ] 

Simon Helsen commented on JENA-308:
---

According to 
http://mail-archives.apache.org/mod_mbox/jena-dev/201208.mbox/%3C503F2C80.1040303%40apache.org%3E,
 the exception com.hp.hpl.jena.tdb.base.block.BlockException: BlockAccessBase: 
Bounds exception: C:\Temp\testIndexLocation\GSPO.dat: (9,9) has been fixed in 
0.9.4, which corresponds to my observations. However, the other 2 exceptions 
are still observable in 0.9.4

> Index corruption after killing process
> --
>
> Key: JENA-308
> URL: https://issues.apache.org/jira/browse/JENA-308
> Project: Apache Jena
>  Issue Type: Bug
>  Components: TDB
>Affects Versions: TDB 0.9.1, TDB 0.9.4
> Environment: Windows 2008/7 (64 bit) both on IBM and SUN JVM 6. 
> Direct mode
>Reporter: Simon Helsen
>Priority: Critical
> Attachments: journalRecoveryTest.zip
>
>
> We are faced with a series of possible exceptions which may or may not occur 
> after an active tdb store is killed. The most common exception has the form:
> Exception in thread "Thread-3" 
> com.hp.hpl.jena.tdb.transaction.TDBTransactionException: Feaild to read block 
> checksum.
>   at com.hp.hpl.jena.tdb.transaction.Journal._read(Journal.java:238)
>   at 
> com.hp.hpl.jena.tdb.transaction.Journal._readJournal(Journal.java:197)
>   at com.hp.hpl.jena.tdb.transaction.Journal.access$1(Journal.java:192)
>   at 
> com.hp.hpl.jena.tdb.transaction.Journal$IteratorEntries.moveToNext(Journal.java:267)
>   at 
> com.hp.hpl.jena.tdb.transaction.Journal$IteratorEntries.moveToNext(Journal.java:1)
>   at 
> org.openjena.atlas.iterator.IteratorSlotted.hasNext(IteratorSlotted.java:67)
>   at 
> com.hp.hpl.jena.tdb.transaction.JournalControl.scanForCommit(JournalControl.java:148)
>   at 
> com.hp.hpl.jena.tdb.transaction.JournalControl.recoverFromJournal(JournalControl.java:127)
>   at 
> com.hp.hpl.jena.tdb.StoreConnection._makeAndCache(StoreConnection.java:234)
>   at com.hp.hpl.jena.tdb.StoreConnection.make(StoreConnection.java:214)
>   at 
> com.hp.hpl.jena.tdb.transaction.DatasetGraphTransaction.(DatasetGraphTransaction.java:76)
>   at com.hp.hpl.jena.tdb.sys.TDBMaker._create(TDBMaker.java:57)
>   at 
> com.hp.hpl.jena.tdb.sys.TDBMaker.createDatasetGraphTransaction(TDBMaker.java:45)
>   at 
> com.hp.hpl.jena.tdb.TDBFactory._createDatasetGraph(TDBFactory.java:104)
>   at com.hp.hpl.jena.tdb.TDBFactory.createDatasetGraph(TDBFactory.java:73)
>   at com.hp.hpl.jena.tdb.TDBFactory.createDataset(TDBFactory.java:52)
> Other exceptions are possible (see comments below). I have been able to 
> produce a standalone java program/test case which simulates a process kill 
> and is able to produce the exception after a non-deterministic time. It runs 
> an iteration until it hits one of the possible problems. Sometimes, an 
> exception hits after just 2 iterations, other times, it can take 40 runs or 
> more before hitting a problem. Because I don't know what the cause is here, I 
> have not been able to make the test case more specific

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (JENA-308) Index corruption after killing process

[Simon Helsen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/JENA-308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Simon Helsen updated JENA-308:
--

Attachment: journalRecoveryTest.zip

> Index corruption after killing process
> --
>
> Key: JENA-308
> URL: https://issues.apache.org/jira/browse/JENA-308
> Project: Apache Jena
>  Issue Type: Bug
>  Components: TDB
>Affects Versions: TDB 0.9.1, TDB 0.9.4
> Environment: Windows 2008/7 (64 bit) both on IBM and SUN JVM 6. 
> Direct mode
>Reporter: Simon Helsen
>Priority: Critical
> Attachments: journalRecoveryTest.zip
>
>
> We are faced with a series of possible exceptions which may or may not occur 
> after an active tdb store is killed. The most common exception has the form:
> Exception in thread "Thread-3" 
> com.hp.hpl.jena.tdb.transaction.TDBTransactionException: Feaild to read block 
> checksum.
>   at com.hp.hpl.jena.tdb.transaction.Journal._read(Journal.java:238)
>   at 
> com.hp.hpl.jena.tdb.transaction.Journal._readJournal(Journal.java:197)
>   at com.hp.hpl.jena.tdb.transaction.Journal.access$1(Journal.java:192)
>   at 
> com.hp.hpl.jena.tdb.transaction.Journal$IteratorEntries.moveToNext(Journal.java:267)
>   at 
> com.hp.hpl.jena.tdb.transaction.Journal$IteratorEntries.moveToNext(Journal.java:1)
>   at 
> org.openjena.atlas.iterator.IteratorSlotted.hasNext(IteratorSlotted.java:67)
>   at 
> com.hp.hpl.jena.tdb.transaction.JournalControl.scanForCommit(JournalControl.java:148)
>   at 
> com.hp.hpl.jena.tdb.transaction.JournalControl.recoverFromJournal(JournalControl.java:127)
>   at 
> com.hp.hpl.jena.tdb.StoreConnection._makeAndCache(StoreConnection.java:234)
>   at com.hp.hpl.jena.tdb.StoreConnection.make(StoreConnection.java:214)
>   at 
> com.hp.hpl.jena.tdb.transaction.DatasetGraphTransaction.(DatasetGraphTransaction.java:76)
>   at com.hp.hpl.jena.tdb.sys.TDBMaker._create(TDBMaker.java:57)
>   at 
> com.hp.hpl.jena.tdb.sys.TDBMaker.createDatasetGraphTransaction(TDBMaker.java:45)
>   at 
> com.hp.hpl.jena.tdb.TDBFactory._createDatasetGraph(TDBFactory.java:104)
>   at com.hp.hpl.jena.tdb.TDBFactory.createDatasetGraph(TDBFactory.java:73)
>   at com.hp.hpl.jena.tdb.TDBFactory.createDataset(TDBFactory.java:52)
> Other exceptions are possible (see comments below). I have been able to 
> produce a standalone java program/test case which simulates a process kill 
> and is able to produce the exception after a non-deterministic time. It runs 
> an iteration until it hits one of the possible problems. Sometimes, an 
> exception hits after just 2 iterations, other times, it can take 40 runs or 
> more before hitting a problem. Because I don't know what the cause is here, I 
> have not been able to make the test case more specific

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-308) Index corruption after killing process

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1393#comment-1393
 ] 

Simon Helsen commented on JENA-308:
---

Finally, I should make a few comments about the actual test case. Because we 
need to kill ongoing (write) transactions and repeat the process often enough, 
I decided to simulate this by running the actual logic in a separate thread 
which I would kill after a certain time. To make sure that no cleanup 
activities take place, I use ThreadDeath to immediately leave the process. 

Moreover, because I want to simulate a fresh startup, I have to call 
StoreConnection.reset();

I realize this is not a 100% perfect simulation, I think the relevant behavior 
is similar to what would happen on process kill. Either way, we have seen the 
exceptions with a real process kill

> Index corruption after killing process
> --
>
> Key: JENA-308
> URL: https://issues.apache.org/jira/browse/JENA-308
> Project: Apache Jena
>  Issue Type: Bug
>  Components: TDB
>Affects Versions: TDB 0.9.1, TDB 0.9.4
> Environment: Windows 2008/7 (64 bit) both on IBM and SUN JVM 6. 
> Direct mode
>Reporter: Simon Helsen
>Priority: Critical
>
> We are faced with a series of possible exceptions which may or may not occur 
> after an active tdb store is killed. The most common exception has the form:
> Exception in thread "Thread-3" 
> com.hp.hpl.jena.tdb.transaction.TDBTransactionException: Feaild to read block 
> checksum.
>   at com.hp.hpl.jena.tdb.transaction.Journal._read(Journal.java:238)
>   at 
> com.hp.hpl.jena.tdb.transaction.Journal._readJournal(Journal.java:197)
>   at com.hp.hpl.jena.tdb.transaction.Journal.access$1(Journal.java:192)
>   at 
> com.hp.hpl.jena.tdb.transaction.Journal$IteratorEntries.moveToNext(Journal.java:267)
>   at 
> com.hp.hpl.jena.tdb.transaction.Journal$IteratorEntries.moveToNext(Journal.java:1)
>   at 
> org.openjena.atlas.iterator.IteratorSlotted.hasNext(IteratorSlotted.java:67)
>   at 
> com.hp.hpl.jena.tdb.transaction.JournalControl.scanForCommit(JournalControl.java:148)
>   at 
> com.hp.hpl.jena.tdb.transaction.JournalControl.recoverFromJournal(JournalControl.java:127)
>   at 
> com.hp.hpl.jena.tdb.StoreConnection._makeAndCache(StoreConnection.java:234)
>   at com.hp.hpl.jena.tdb.StoreConnection.make(StoreConnection.java:214)
>   at 
> com.hp.hpl.jena.tdb.transaction.DatasetGraphTransaction.(DatasetGraphTransaction.java:76)
>   at com.hp.hpl.jena.tdb.sys.TDBMaker._create(TDBMaker.java:57)
>   at 
> com.hp.hpl.jena.tdb.sys.TDBMaker.createDatasetGraphTransaction(TDBMaker.java:45)
>   at 
> com.hp.hpl.jena.tdb.TDBFactory._createDatasetGraph(TDBFactory.java:104)
>   at com.hp.hpl.jena.tdb.TDBFactory.createDatasetGraph(TDBFactory.java:73)
>   at com.hp.hpl.jena.tdb.TDBFactory.createDataset(TDBFactory.java:52)
> Other exceptions are possible (see comments below). I have been able to 
> produce a standalone java program/test case which simulates a process kill 
> and is able to produce the exception after a non-deterministic time. It runs 
> an iteration until it hits one of the possible problems. Sometimes, an 
> exception hits after just 2 iterations, other times, it can take 40 runs or 
> more before hitting a problem. Because I don't know what the cause is here, I 
> have not been able to make the test case more specific

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-308) Index corruption after killing process

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1389#comment-1389
 ] 

Simon Helsen commented on JENA-308:
---

in 0.9.1 (not yet seen in 0.9.4), we also sometimes see:

1346278896399 : Iteration 10
1346278896400 : create dataset
Exception in thread "Thread-11" com.hp.hpl.jena.tdb.base.block.BlockException: 
BlockAccessBase: Bounds exception: C:\Temp\testIndexLocation\GSPO.dat: (9,9)
at 
com.hp.hpl.jena.tdb.base.file.BlockAccessBase.check(BlockAccessBase.java:107)
at 
com.hp.hpl.jena.tdb.base.file.BlockAccessBase.check(BlockAccessBase.java:114)
at 
com.hp.hpl.jena.tdb.base.file.BlockAccessDirect.write(BlockAccessDirect.java:85)
at 
com.hp.hpl.jena.tdb.base.file.BlockAccessDirect.overwrite(BlockAccessDirect.java:104)
at 
com.hp.hpl.jena.tdb.base.block.BlockMgrFileAccess.overwrite(BlockMgrFileAccess.java:102)
at 
com.hp.hpl.jena.tdb.base.block.BlockMgrWrapper.overwrite(BlockMgrWrapper.java:89)
at 
com.hp.hpl.jena.tdb.base.block.BlockMgrSync.overwrite(BlockMgrSync.java:90)
at 
com.hp.hpl.jena.tdb.base.block.BlockMgrCache.overwrite(BlockMgrCache.java:206)
at 
com.hp.hpl.jena.tdb.transaction.JournalControl.replay(JournalControl.java:305)
at 
com.hp.hpl.jena.tdb.transaction.JournalControl.recoverSegment(JournalControl.java:175)
at 
com.hp.hpl.jena.tdb.transaction.JournalControl.recoverFromJournal(JournalControl.java:128)
at 
com.hp.hpl.jena.tdb.StoreConnection._makeAndCache(StoreConnection.java:231)
at com.hp.hpl.jena.tdb.StoreConnection.make(StoreConnection.java:190)
at 
com.hp.hpl.jena.tdb.transaction.DatasetGraphTransaction.(DatasetGraphTransaction.java:63)
at com.hp.hpl.jena.tdb.sys.TDBMakerTxn._create(TDBMakerTxn.java:49)
at 
com.hp.hpl.jena.tdb.sys.TDBMakerTxn.createDatasetGraph(TDBMakerTxn.java:37)
at 
com.hp.hpl.jena.tdb.TDBFactory._createDatasetGraph(TDBFactory.java:104)
at com.hp.hpl.jena.tdb.TDBFactory.createDatasetGraph(TDBFactory.java:73)
at com.hp.hpl.jena.tdb.TDBFactory.createDataset(TDBFactory.java:52)
at 
com.ibm.jena.test.JournalRecoveryTest$TestRunnable.run(JournalRecoveryTest.java:161)
at java.lang.Thread.run(Thread.java:662)

This was the original exception which triggered my investigation. It was 
observed by one of our clients after a windows 2008 server machine was rebooted 
killing the Java process. It is unclear if this issues has disappeared in 0.9.4

> Index corruption after killing process
> --
>
> Key: JENA-308
> URL: https://issues.apache.org/jira/browse/JENA-308
> Project: Apache Jena
>  Issue Type: Bug
>  Components: TDB
>Affects Versions: TDB 0.9.1, TDB 0.9.4
> Environment: Windows 2008/7 (64 bit) both on IBM and SUN JVM 6. 
> Direct mode
>Reporter: Simon Helsen
>Priority: Critical
>
> We are faced with a series of possible exceptions which may or may not occur 
> after an active tdb store is killed. The most common exception has the form:
> Exception in thread "Thread-3" 
> com.hp.hpl.jena.tdb.transaction.TDBTransactionException: Feaild to read block 
> checksum.
>   at com.hp.hpl.jena.tdb.transaction.Journal._read(Journal.java:238)
>   at 
> com.hp.hpl.jena.tdb.transaction.Journal._readJournal(Journal.java:197)
>   at com.hp.hpl.jena.tdb.transaction.Journal.access$1(Journal.java:192)
>   at 
> com.hp.hpl.jena.tdb.transaction.Journal$IteratorEntries.moveToNext(Journal.java:267)
>   at 
> com.hp.hpl.jena.tdb.transaction.Journal$IteratorEntries.moveToNext(Journal.java:1)
>   at 
> org.openjena.atlas.iterator.IteratorSlotted.hasNext(IteratorSlotted.java:67)
>   at 
> com.hp.hpl.jena.tdb.transaction.JournalControl.scanForCommit(JournalControl.java:148)
>   at 
> com.hp.hpl.jena.tdb.transaction.JournalControl.recoverFromJournal(JournalControl.java:127)
>   at 
> com.hp.hpl.jena.tdb.StoreConnection._makeAndCache(StoreConnection.java:234)
>   at com.hp.hpl.jena.tdb.StoreConnection.make(StoreConnection.java:214)
>   at 
> com.hp.hpl.jena.tdb.transaction.DatasetGraphTransaction.(DatasetGraphTransaction.java:76)
>   at com.hp.hpl.jena.tdb.sys.TDBMaker._create(TDBMaker.java:57)
>   at 
> com.hp.hpl.jena.tdb.sys.TDBMaker.createDatasetGraphTransaction(TDBMaker.java:45)
>   at 
> com.hp.hpl.jena.tdb.TDBFactory._createDatasetGraph(TDBFactory.java:104)
>   at com.hp.hpl.jena.tdb.TDBFactory.createDatasetGraph(TDBFactory.java:73)
>   at com.hp.hpl.jena.tdb.TDBFactory.createDataset(TDBFactory.java:52)
> Other exceptions are possible (see comments below). I have been able to 
> produce a standalone java program/test case which simulates a process kill 
> and is able to produce the exception af

[jira] [Commented] (JENA-308) Index corruption after killing process

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1388#comment-1388
 ] 

Simon Helsen commented on JENA-308:
---

The exception as shown in the description is the most common and has been 
observed on both 0.9.1 and 0.9.4. However, the following exception very rarely 
occurs instead (also seen in both 0.9.1. and 0.9.4):

1346278257925 : Iteration 1
1346278257931 : create dataset
* UNEXPECTED [1]
1346278257945 : read
1346278257947 : write
18:10:57,969 [Thread-2]  WARN com.hp.hpl.jena.tdb.transaction.NodeTableTrans
  - Txn[4]/W journalStartOffset not zero: 1048/0x418


Different ids for https://localhost:9443/jazz/storage/net.jazz.ajax.theme0: 
allocated: expected [0665], got []
>>
label = nodes
txn = Transaction: 4 : Mode=WRITE : State=PREPARING : C:\Temp\testIndexLocation\
offset = 1637
journalStartOffset = 1048
journal = nodes.dat-jrnl

com.hp.hpl.jena.tdb.TDBException: Different ids for 
https://localhost:9443/jazz/storage/net.jazz.ajax.theme0: allocated: expected 
[0665], got []
at 
com.hp.hpl.jena.tdb.transaction.NodeTableTrans.inconsistent(NodeTableTrans.java:212)
at 
com.hp.hpl.jena.tdb.transaction.NodeTableTrans.append(NodeTableTrans.java:200)
at 
com.hp.hpl.jena.tdb.transaction.NodeTableTrans.writeNodeJournal(NodeTableTrans.java:306)
at 
com.hp.hpl.jena.tdb.transaction.NodeTableTrans.commitPrepare(NodeTableTrans.java:266)
at 
com.hp.hpl.jena.tdb.transaction.Transaction.prepare(Transaction.java:131)
at 
com.hp.hpl.jena.tdb.transaction.Transaction.commit(Transaction.java:112)
at 
com.hp.hpl.jena.tdb.transaction.DatasetGraphTxn.commit(DatasetGraphTxn.java:44)
at 
com.hp.hpl.jena.tdb.transaction.DatasetGraphTransaction._commit(DatasetGraphTransaction.java:148)
at 
com.hp.hpl.jena.sparql.core.DatasetGraphTrackActive.commit(DatasetGraphTrackActive.java:54)
at com.hp.hpl.jena.sparql.core.DatasetImpl.commit(DatasetImpl.java:141)
at 
com.ibm.jena.test.JournalRecoveryTest.storeOperation(JournalRecoveryTest.java:92)
at 
com.ibm.jena.test.JournalRecoveryTest.writeModel(JournalRecoveryTest.java:212)
at 
com.ibm.jena.test.JournalRecoveryTest$TestRunnable.run(JournalRecoveryTest.java:277)
at java.lang.Thread.run(Thread.java:662)

> Index corruption after killing process
> --
>
> Key: JENA-308
> URL: https://issues.apache.org/jira/browse/JENA-308
> Project: Apache Jena
>  Issue Type: Bug
>  Components: TDB
>Affects Versions: TDB 0.9.1, TDB 0.9.4
> Environment: Windows 2008/7 (64 bit) both on IBM and SUN JVM 6. 
> Direct mode
>Reporter: Simon Helsen
>Priority: Critical
>
> We are faced with a series of possible exceptions which may or may not occur 
> after an active tdb store is killed. The most common exception has the form:
> Exception in thread "Thread-3" 
> com.hp.hpl.jena.tdb.transaction.TDBTransactionException: Feaild to read block 
> checksum.
>   at com.hp.hpl.jena.tdb.transaction.Journal._read(Journal.java:238)
>   at 
> com.hp.hpl.jena.tdb.transaction.Journal._readJournal(Journal.java:197)
>   at com.hp.hpl.jena.tdb.transaction.Journal.access$1(Journal.java:192)
>   at 
> com.hp.hpl.jena.tdb.transaction.Journal$IteratorEntries.moveToNext(Journal.java:267)
>   at 
> com.hp.hpl.jena.tdb.transaction.Journal$IteratorEntries.moveToNext(Journal.java:1)
>   at 
> org.openjena.atlas.iterator.IteratorSlotted.hasNext(IteratorSlotted.java:67)
>   at 
> com.hp.hpl.jena.tdb.transaction.JournalControl.scanForCommit(JournalControl.java:148)
>   at 
> com.hp.hpl.jena.tdb.transaction.JournalControl.recoverFromJournal(JournalControl.java:127)
>   at 
> com.hp.hpl.jena.tdb.StoreConnection._makeAndCache(StoreConnection.java:234)
>   at com.hp.hpl.jena.tdb.StoreConnection.make(StoreConnection.java:214)
>   at 
> com.hp.hpl.jena.tdb.transaction.DatasetGraphTransaction.(DatasetGraphTransaction.java:76)
>   at com.hp.hpl.jena.tdb.sys.TDBMaker._create(TDBMaker.java:57)
>   at 
> com.hp.hpl.jena.tdb.sys.TDBMaker.createDatasetGraphTransaction(TDBMaker.java:45)
>   at 
> com.hp.hpl.jena.tdb.TDBFactory._createDatasetGraph(TDBFactory.java:104)
>   at com.hp.hpl.jena.tdb.TDBFactory.createDatasetGraph(TDBFactory.java:73)
>   at com.hp.hpl.jena.tdb.TDBFactory.createDataset(TDBFactory.java:52)
> Other exceptions are possible (see comments below). I have been able to 
> produce a standalone java program/test case which simulates a process kill 
> and is able to produce the exception after a non-deterministic time. It runs 
> an iteration until it hits one of the possible problems. Sometimes, an 
> exception hits 

[jira] [Created] (JENA-308) Index corruption after killing process

[Simon Helsen (JIRA)]

Simon Helsen created JENA-308:
-

 Summary: Index corruption after killing process
 Key: JENA-308
 URL: https://issues.apache.org/jira/browse/JENA-308
 Project: Apache Jena
  Issue Type: Bug
  Components: TDB
Affects Versions: TDB 0.9.1, TDB 0.9.4
 Environment: Windows 2008/7 (64 bit) both on IBM and SUN JVM 6. Direct 
mode
Reporter: Simon Helsen
Priority: Critical


We are faced with a series of possible exceptions which may or may not occur 
after an active tdb store is killed. The most common exception has the form:

Exception in thread "Thread-3" 
com.hp.hpl.jena.tdb.transaction.TDBTransactionException: Feaild to read block 
checksum.
at com.hp.hpl.jena.tdb.transaction.Journal._read(Journal.java:238)
at 
com.hp.hpl.jena.tdb.transaction.Journal._readJournal(Journal.java:197)
at com.hp.hpl.jena.tdb.transaction.Journal.access$1(Journal.java:192)
at 
com.hp.hpl.jena.tdb.transaction.Journal$IteratorEntries.moveToNext(Journal.java:267)
at 
com.hp.hpl.jena.tdb.transaction.Journal$IteratorEntries.moveToNext(Journal.java:1)
at 
org.openjena.atlas.iterator.IteratorSlotted.hasNext(IteratorSlotted.java:67)
at 
com.hp.hpl.jena.tdb.transaction.JournalControl.scanForCommit(JournalControl.java:148)
at 
com.hp.hpl.jena.tdb.transaction.JournalControl.recoverFromJournal(JournalControl.java:127)
at 
com.hp.hpl.jena.tdb.StoreConnection._makeAndCache(StoreConnection.java:234)
at com.hp.hpl.jena.tdb.StoreConnection.make(StoreConnection.java:214)
at 
com.hp.hpl.jena.tdb.transaction.DatasetGraphTransaction.(DatasetGraphTransaction.java:76)
at com.hp.hpl.jena.tdb.sys.TDBMaker._create(TDBMaker.java:57)
at 
com.hp.hpl.jena.tdb.sys.TDBMaker.createDatasetGraphTransaction(TDBMaker.java:45)
at 
com.hp.hpl.jena.tdb.TDBFactory._createDatasetGraph(TDBFactory.java:104)
at com.hp.hpl.jena.tdb.TDBFactory.createDatasetGraph(TDBFactory.java:73)
at com.hp.hpl.jena.tdb.TDBFactory.createDataset(TDBFactory.java:52)

Other exceptions are possible (see comments below). I have been able to produce 
a standalone java program/test case which simulates a process kill and is able 
to produce the exception after a non-deterministic time. It runs an iteration 
until it hits one of the possible problems. Sometimes, an exception hits after 
just 2 iterations, other times, it can take 40 runs or more before hitting a 
problem. Because I don't know what the cause is here, I have not been able to 
make the test case more specific

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-301) RecordRangeIterator: records not strictly increasing

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13439654#comment-13439654
 ] 

Simon Helsen commented on JENA-301:
---

So, with updating to the overnight changes, I ran this test in mapped mode and 
I was not seeing the same problem which I saw in direct mode. I then updated 
everything to HEAD and ran the tests again and this time, also in direct mode I 
was not seeing the problem anymore. So, it seems to me this issue is resolved 
wit the latest changes.

> RecordRangeIterator: records not strictly increasing
> 
>
> Key: JENA-301
> URL: https://issues.apache.org/jira/browse/JENA-301
> Project: Apache Jena
>  Issue Type: Bug
>  Components: TDB
>Affects Versions: TDB 0.9.4
> Environment: windows 64 bit, oracle JRE 6
>Reporter: Simon Helsen
> Attachments: PerformanceRegressionTest.java
>
>
> When I tried to execute the provided test case for JENA-256, I ran into a TDB 
> Storage Exception. The scenario is like this
> 1) run the test application on a clean non-existing index. All should go 
> well, i.e. not exceptions, tests finish fine
> 2) then run the test application again on the same index. You'll end up with 
> the following stack trace at write operation 98:
> xception in thread "main" java.lang.RuntimeException: 
> com.hp.hpl.jena.tdb.base.StorageException: RecordRangeIterator: records not 
> strictly increasing: 
> 20ba002401490ad3 // 
> 08bc002401490271
>   at 
> com.ibm.jena.test.PerformanceRegressionTest.main(PerformanceRegressionTest.java:101)
> Caused by: com.hp.hpl.jena.tdb.base.StorageException: RecordRangeIterator: 
> records not strictly increasing: 
> 20ba002401490ad3 // 
> 08bc002401490271
>   at 
> com.hp.hpl.jena.tdb.base.recordbuffer.RecordRangeIterator.hasNext(RecordRangeIterator.java:124)
>   at org.openjena.atlas.iterator.Iter$4.hasNext(Iter.java:295)
>   at 
> com.hp.hpl.jena.tdb.sys.DatasetControlMRSW$IteratorCheckNotConcurrent.hasNext(DatasetControlMRSW.java:119)
>   at 
> com.hp.hpl.jena.tdb.graph.BulkUpdateHandlerTDB.removeWorker(BulkUpdateHandlerTDB.java:142)
>   at 
> com.hp.hpl.jena.tdb.graph.BulkUpdateHandlerTDB.removeAll(BulkUpdateHandlerTDB.java:102)
>   at com.hp.hpl.jena.rdf.model.impl.ModelCom.removeAll(ModelCom.java:372)
>   at 
> com.ibm.jena.test.PerformanceRegressionTest.main(PerformanceRegressionTest.java:78)
> The issue seems reproducible (I have seen it 3 times in a row when following 
> the scenario as described). Although I think this is exactly the same test 
> application as in JENA-256, I'll add it again for completeness

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-299) LeaveCriticalSection Error

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13438995#comment-13438995
 ] 

Simon Helsen commented on JENA-299:
---

yes, I agree, still trying to produce a small independent test case to 
reproduce the "Quad: predicate cannot be null"

> LeaveCriticalSection Error
> --
>
> Key: JENA-299
> URL: https://issues.apache.org/jira/browse/JENA-299
> Project: Apache Jena
>  Issue Type: Bug
>  Components: TDB
>Affects Versions: TDB 0.9.4
> Environment: windows 64 bit
>Reporter: Simon Helsen
>Assignee: Andy Seaborne
>Priority: Critical
> Fix For: TDB 0.9.4
>
> Attachments: LeaveCriticalSectionTestCase.zip
>
>
> I have attached a standalone test case, which, when run with the latest 
> snapshot produces the following exception:
> Exception in thread "main" com.hp.hpl.jena.shared.JenaException: 
> leaveCriticalSection: No lock held (main) Thread R/W: 0/0 :: Model R/W: 0/0 
> (thread: main)
>   at 
> com.hp.hpl.jena.shared.LockMRSW.leaveCriticalSection(LockMRSW.java:175)
>   at 
> com.hp.hpl.jena.tdb.transaction.TransactionManager$TSM_WriteBackEndTxn.readerFinishes(TransactionManager.java:210)
>   at 
> com.hp.hpl.jena.tdb.transaction.TransactionManager.readerFinishes(TransactionManager.java:723)
>   at 
> com.hp.hpl.jena.tdb.transaction.TransactionManager.noteTxnAbort(TransactionManager.java:587)
>   at 
> com.hp.hpl.jena.tdb.transaction.TransactionManager.notifyAbort(TransactionManager.java:445)
>   at 
> com.hp.hpl.jena.tdb.transaction.Transaction.abort(Transaction.java:162)
>   at 
> com.hp.hpl.jena.tdb.transaction.DatasetGraphTxn.abort(DatasetGraphTxn.java:45)
>   at 
> com.hp.hpl.jena.tdb.transaction.DatasetGraphTransaction._abort(DatasetGraphTransaction.java:156)
>   at 
> com.hp.hpl.jena.tdb.migrate.DatasetGraphTrackActive.abort(DatasetGraphTrackActive.java:68)
>   at com.hp.hpl.jena.sparql.core.DatasetImpl.abort(DatasetImpl.java:149)
>   at 
> com.ibm.jena.test.LeaveCriticalSectionErrorTest.storeOperation(LeaveCriticalSectionErrorTest.java:57)
>   at 
> com.ibm.jena.test.LeaveCriticalSectionErrorTest.query1(LeaveCriticalSectionErrorTest.java:105)
>   at 
> com.ibm.jena.test.LeaveCriticalSectionErrorTest.main(LeaveCriticalSectionErrorTest.java:156)
> The sequence in the test case is to run 2 queries, 1 write and then again 2 
> queries:
>   test.query1();
>   test.query1();
>   test.write1();
>   test.query1();
>   test.query1();
> Somehow, the following sequence did not produce the exception:
>   test.query1();
>   test.query1();
>   test.write1();
>   test.query1();
> Note that the test case does not check the correctness of any results.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-299) LeaveCriticalSection Error

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13438986#comment-13438986
 ] 

Simon Helsen commented on JENA-299:
---

just after I wrote this I noticed it is bound to the debug level, so ignore my 
question

> LeaveCriticalSection Error
> --
>
> Key: JENA-299
> URL: https://issues.apache.org/jira/browse/JENA-299
> Project: Apache Jena
>  Issue Type: Bug
>  Components: TDB
>Affects Versions: TDB 0.9.4
> Environment: windows 64 bit
>Reporter: Simon Helsen
>Assignee: Andy Seaborne
>Priority: Critical
> Fix For: TDB 0.9.4
>
> Attachments: LeaveCriticalSectionTestCase.zip
>
>
> I have attached a standalone test case, which, when run with the latest 
> snapshot produces the following exception:
> Exception in thread "main" com.hp.hpl.jena.shared.JenaException: 
> leaveCriticalSection: No lock held (main) Thread R/W: 0/0 :: Model R/W: 0/0 
> (thread: main)
>   at 
> com.hp.hpl.jena.shared.LockMRSW.leaveCriticalSection(LockMRSW.java:175)
>   at 
> com.hp.hpl.jena.tdb.transaction.TransactionManager$TSM_WriteBackEndTxn.readerFinishes(TransactionManager.java:210)
>   at 
> com.hp.hpl.jena.tdb.transaction.TransactionManager.readerFinishes(TransactionManager.java:723)
>   at 
> com.hp.hpl.jena.tdb.transaction.TransactionManager.noteTxnAbort(TransactionManager.java:587)
>   at 
> com.hp.hpl.jena.tdb.transaction.TransactionManager.notifyAbort(TransactionManager.java:445)
>   at 
> com.hp.hpl.jena.tdb.transaction.Transaction.abort(Transaction.java:162)
>   at 
> com.hp.hpl.jena.tdb.transaction.DatasetGraphTxn.abort(DatasetGraphTxn.java:45)
>   at 
> com.hp.hpl.jena.tdb.transaction.DatasetGraphTransaction._abort(DatasetGraphTransaction.java:156)
>   at 
> com.hp.hpl.jena.tdb.migrate.DatasetGraphTrackActive.abort(DatasetGraphTrackActive.java:68)
>   at com.hp.hpl.jena.sparql.core.DatasetImpl.abort(DatasetImpl.java:149)
>   at 
> com.ibm.jena.test.LeaveCriticalSectionErrorTest.storeOperation(LeaveCriticalSectionErrorTest.java:57)
>   at 
> com.ibm.jena.test.LeaveCriticalSectionErrorTest.query1(LeaveCriticalSectionErrorTest.java:105)
>   at 
> com.ibm.jena.test.LeaveCriticalSectionErrorTest.main(LeaveCriticalSectionErrorTest.java:156)
> The sequence in the test case is to run 2 queries, 1 write and then again 2 
> queries:
>   test.query1();
>   test.query1();
>   test.write1();
>   test.query1();
>   test.query1();
> Somehow, the following sequence did not produce the exception:
>   test.query1();
>   test.query1();
>   test.write1();
>   test.query1();
> Note that the test case does not check the correctness of any results.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-299) LeaveCriticalSection Error

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13438985#comment-13438985
 ] 

Simon Helsen commented on JENA-299:
---

Andy, to follow-up on your comment

"Logging: turn the system log "TDB" or the TransactionManager class log. Either 
will cause the transaction cycle to be logged."

I looked at that but I don't see how this would do what I want. There is

 private static Logger log = LoggerFactory.getLogger(TransactionManager.class) ;

but it only reports errors and warnings.

I also noticed 

  // Record happenings.
private boolean recordHistory = false ;

Is that what you meant? If so, could that be controlled by log4j as well? (e.g. 
as trace-level?) I don't like to have to change Jena code because in our 
production system, I can't reach that short of patching the binary jar




> LeaveCriticalSection Error
> --
>
> Key: JENA-299
> URL: https://issues.apache.org/jira/browse/JENA-299
> Project: Apache Jena
>  Issue Type: Bug
>  Components: TDB
>Affects Versions: TDB 0.9.4
> Environment: windows 64 bit
>Reporter: Simon Helsen
>Assignee: Andy Seaborne
>Priority: Critical
> Fix For: TDB 0.9.4
>
> Attachments: LeaveCriticalSectionTestCase.zip
>
>
> I have attached a standalone test case, which, when run with the latest 
> snapshot produces the following exception:
> Exception in thread "main" com.hp.hpl.jena.shared.JenaException: 
> leaveCriticalSection: No lock held (main) Thread R/W: 0/0 :: Model R/W: 0/0 
> (thread: main)
>   at 
> com.hp.hpl.jena.shared.LockMRSW.leaveCriticalSection(LockMRSW.java:175)
>   at 
> com.hp.hpl.jena.tdb.transaction.TransactionManager$TSM_WriteBackEndTxn.readerFinishes(TransactionManager.java:210)
>   at 
> com.hp.hpl.jena.tdb.transaction.TransactionManager.readerFinishes(TransactionManager.java:723)
>   at 
> com.hp.hpl.jena.tdb.transaction.TransactionManager.noteTxnAbort(TransactionManager.java:587)
>   at 
> com.hp.hpl.jena.tdb.transaction.TransactionManager.notifyAbort(TransactionManager.java:445)
>   at 
> com.hp.hpl.jena.tdb.transaction.Transaction.abort(Transaction.java:162)
>   at 
> com.hp.hpl.jena.tdb.transaction.DatasetGraphTxn.abort(DatasetGraphTxn.java:45)
>   at 
> com.hp.hpl.jena.tdb.transaction.DatasetGraphTransaction._abort(DatasetGraphTransaction.java:156)
>   at 
> com.hp.hpl.jena.tdb.migrate.DatasetGraphTrackActive.abort(DatasetGraphTrackActive.java:68)
>   at com.hp.hpl.jena.sparql.core.DatasetImpl.abort(DatasetImpl.java:149)
>   at 
> com.ibm.jena.test.LeaveCriticalSectionErrorTest.storeOperation(LeaveCriticalSectionErrorTest.java:57)
>   at 
> com.ibm.jena.test.LeaveCriticalSectionErrorTest.query1(LeaveCriticalSectionErrorTest.java:105)
>   at 
> com.ibm.jena.test.LeaveCriticalSectionErrorTest.main(LeaveCriticalSectionErrorTest.java:156)
> The sequence in the test case is to run 2 queries, 1 write and then again 2 
> queries:
>   test.query1();
>   test.query1();
>   test.write1();
>   test.query1();
>   test.query1();
> Somehow, the following sequence did not produce the exception:
>   test.query1();
>   test.query1();
>   test.write1();
>   test.query1();
> Note that the test case does not check the correctness of any results.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (JENA-294) TransformFilterEquality does not handle starting OPTIONAL well

[Simon Helsen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/JENA-294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Simon Helsen updated JENA-294:
--

Attachment: manifestPatchForOptFilterEquality.txt
filter-equality-13.srj
filter-equality-13.rq

for completeness and because you asked.

> TransformFilterEquality does not handle starting OPTIONAL well
> --
>
> Key: JENA-294
> URL: https://issues.apache.org/jira/browse/JENA-294
> Project: Apache Jena
>  Issue Type: Improvement
>  Components: ARQ
>Affects Versions: Jena 2.7.4
>Reporter: Simon Helsen
>Assignee: Andy Seaborne
> Fix For: ARQ 2.9.4
>
> Attachments: filter-equality-13.rq, filter-equality-13.srj, 
> jena-294-afs-1.patch, manifestPatchForOptFilterEquality.txt, patch.txt, 
> patch.txt, test-filter-equality.zip
>
>
> There was one other case where our tests were stuck on a very slow query 
> execution because transformFilterEquality failed to optimize. The problem is 
> that the optimizer gives up whenever the WHERE clause starts with an OPTIONAL 
> clause. The reason is that the generated algebraic formula starts with a 
> TableUnit and this is not handled correctly. 
> I have attached a patch which fixes the problem

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-294) TransformFilterEquality does not handle starting OPTIONAL well

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13438966#comment-13438966
 ] 

Simon Helsen commented on JENA-294:
---

yes, I saw this. Ok, in this case, it would be trivial since 

SELECT *
{
   ?x :p ?o2
   OPTIONAL { ?x :q ?o }
   FILTER(?x = :x)
} 

is equivalent to 

SELECT *
{
   OPTIONAL { ?x :q ?o }
   FILTER(?x = :x)
   ?x :p ?o2
} 

(I can provide a patch, but does that help here?) 

Ok, I see the tests in TestFilterTransform. I assume the exact test for this 
particular use-case depends a bit on how it is fixed. One possibility (but I 
don't know if that could affect performance in other ways) is to always move 
all non-OPTIONAL constraints before OPTIONAL constraints and then let 
TransformFilterEquality do its normal job. 

> TransformFilterEquality does not handle starting OPTIONAL well
> --
>
> Key: JENA-294
> URL: https://issues.apache.org/jira/browse/JENA-294
> Project: Apache Jena
>  Issue Type: Improvement
>  Components: ARQ
>Affects Versions: Jena 2.7.4
>Reporter: Simon Helsen
>Assignee: Andy Seaborne
> Fix For: ARQ 2.9.4
>
> Attachments: jena-294-afs-1.patch, patch.txt, patch.txt, 
> test-filter-equality.zip
>
>
> There was one other case where our tests were stuck on a very slow query 
> execution because transformFilterEquality failed to optimize. The problem is 
> that the optimizer gives up whenever the WHERE clause starts with an OPTIONAL 
> clause. The reason is that the generated algebraic formula starts with a 
> TableUnit and this is not handled correctly. 
> I have attached a patch which fixes the problem

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-294) TransformFilterEquality does not handle starting OPTIONAL well

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13438956#comment-13438956
 ] 

Simon Helsen commented on JENA-294:
---

While I agree your version may be "more" conventional, we had clients who 
produced the one I provided. Not only that, their "less conventional query" was 
generated by a generator so I am sure they had their reasons. Finally, and this 
is important to us, they had their queries tested for performance on our 
previous adoption of 2.6.x and the move to 2.7.x turned it into a performance 
regression for them. As I have mentioned before, we are talking a few 100ms 
versus many minutes. 

So, in this case, we really need a fix for the query I provided above. 
Otherwise, we are still in the same situation as before I opened this defect 
(which is why I reopened). More generally, I am not sure why the same "trick" 
cannot be applied here. Leave filter as it is and simply push the assignment 
down. Why is that unique to OPTIONAL? 

As for providing tests for jena-arq/testing/ARQ/OptFilterEquality/, I am happy 
to add a test if I have one, but it seems that this test suite is only to check 
query correctness not whether the optimization took place, or am I missing 
something

> TransformFilterEquality does not handle starting OPTIONAL well
> --
>
> Key: JENA-294
> URL: https://issues.apache.org/jira/browse/JENA-294
> Project: Apache Jena
>  Issue Type: Improvement
>  Components: ARQ
>Affects Versions: Jena 2.7.4
>Reporter: Simon Helsen
>Assignee: Andy Seaborne
> Fix For: ARQ 2.9.4
>
> Attachments: jena-294-afs-1.patch, patch.txt, patch.txt, 
> test-filter-equality.zip
>
>
> There was one other case where our tests were stuck on a very slow query 
> execution because transformFilterEquality failed to optimize. The problem is 
> that the optimizer gives up whenever the WHERE clause starts with an OPTIONAL 
> clause. The reason is that the generated algebraic formula starts with a 
> TableUnit and this is not handled correctly. 
> I have attached a patch which fixes the problem

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Reopened] (JENA-294) TransformFilterEquality does not handle starting OPTIONAL well

[Simon Helsen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/JENA-294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Simon Helsen reopened JENA-294:
---


I tested against HEAD and yes, for queries with only OPTIONALs,everything seems 
to work correctly now, but the optimization is lost whenever the query starts 
with OPTIONAL, but also has a non-optional clause. E.g.

PREFIX :  
SELECT *
{
   OPTIONAL { ?x :q ?o }
   FILTER(?x = :x)
   ?x :p ?o2
} 

produces 

(prefix ((: ))
  (filter (= ?x :x)
(sequence
  (conditional
(table unit)
(bgp (triple ?x :q ?o)))
  (bgp (triple ?x :p ?o2)

The assigns are lost

> TransformFilterEquality does not handle starting OPTIONAL well
> --
>
> Key: JENA-294
> URL: https://issues.apache.org/jira/browse/JENA-294
> Project: Apache Jena
>  Issue Type: Improvement
>  Components: ARQ
>Affects Versions: Jena 2.7.4
>Reporter: Simon Helsen
>Assignee: Andy Seaborne
> Fix For: ARQ 2.9.4
>
> Attachments: jena-294-afs-1.patch, patch.txt, patch.txt, 
> test-filter-equality.zip
>
>
> There was one other case where our tests were stuck on a very slow query 
> execution because transformFilterEquality failed to optimize. The problem is 
> that the optimizer gives up whenever the WHERE clause starts with an OPTIONAL 
> clause. The reason is that the generated algebraic formula starts with a 
> TableUnit and this is not handled correctly. 
> I have attached a patch which fixes the problem

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-294) TransformFilterEquality does not handle starting OPTIONAL well

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13438829#comment-13438829
 ] 

Simon Helsen commented on JENA-294:
---

right, the problem is that if the filters with the ?R= pattern are not 
converted to assigns, we end up with potentially countless of returns which is 
why we got the performance breakdown from a few ms to minutes. With the 
assigns, the variable is not expanded and tested each time. What you suggest 
makes sense. You introduce an assign for each occurrence of the variable (deep 
down) and just keep the filter where it was originally to make sure the 
semantics is correct (i.e. as before the optimization). I'll definitely check 
it out

> TransformFilterEquality does not handle starting OPTIONAL well
> --
>
> Key: JENA-294
> URL: https://issues.apache.org/jira/browse/JENA-294
> Project: Apache Jena
>  Issue Type: Improvement
>  Components: ARQ
>Affects Versions: Jena 2.7.4
>Reporter: Simon Helsen
>Assignee: Andy Seaborne
> Fix For: ARQ 2.9.4
>
> Attachments: jena-294-afs-1.patch, patch.txt, patch.txt, 
> test-filter-equality.zip
>
>
> There was one other case where our tests were stuck on a very slow query 
> execution because transformFilterEquality failed to optimize. The problem is 
> that the optimizer gives up whenever the WHERE clause starts with an OPTIONAL 
> clause. The reason is that the generated algebraic formula starts with a 
> TableUnit and this is not handled correctly. 
> I have attached a patch which fixes the problem

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-299) LeaveCriticalSection Error

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13438045#comment-13438045
 ] 

Simon Helsen commented on JENA-299:
---

The issue I reported in this defect is gone. However, I am still observing 
another problem. Whenever I start the product all appears fine, but when I shut 
it down and then start again, I run into the following exception (i.e. some 
sort of corruption). I'll try to produce an independent test case. Perhaps 
there is a relation to JENA-301 as well?

13:44:31,484 [1311264296@qtp-145033381-5] ERROR com.ibm.team.jfs
- @ID@E An unexpected problem occurred while processing a 
transactional model read activity
java.lang.UnsupportedOperationException: Quad: predicate cannot be null
at com.hp.hpl.jena.sparql.core.Quad.(Quad.java:61)
at com.hp.hpl.jena.tdb.lib.TupleLib.quad(TupleLib.java:162)
at com.hp.hpl.jena.tdb.lib.TupleLib.quad(TupleLib.java:153)
at com.hp.hpl.jena.tdb.lib.TupleLib.access$100(TupleLib.java:45)
at com.hp.hpl.jena.tdb.lib.TupleLib$4.convert(TupleLib.java:87)
at com.hp.hpl.jena.tdb.lib.TupleLib$4.convert(TupleLib.java:83)
at org.openjena.atlas.iterator.Iter$4.next(Iter.java:301)
at 
com.hp.hpl.jena.tdb.store.GraphTDBBase$ProjectQuadsToTriples.next(GraphTDBBase.java:178)
at 
com.hp.hpl.jena.tdb.store.GraphTDBBase$ProjectQuadsToTriples.next(GraphTDBBase.java:166)
at 
com.hp.hpl.jena.util.iterator.WrappedIterator.next(WrappedIterator.java:80)
at com.hp.hpl.jena.util.iterator.Map1Iterator.next(Map1Iterator.java:47)
at 
com.hp.hpl.jena.util.iterator.WrappedIterator.next(WrappedIterator.java:80)
at 
com.hp.hpl.jena.rdf.model.impl.StmtIteratorImpl.next(StmtIteratorImpl.java:45)
at 
com.hp.hpl.jena.rdf.model.impl.StmtIteratorImpl.next(StmtIteratorImpl.java:33)
at 
com.ibm.team.jfs.rdf.internal.jena.tdb.JenaTxTdbProvider$7.run(JenaTxTdbProvider.java:950)
at 
com.ibm.team.jfs.rdf.internal.jena.tdb.JenaTxTdbProvider$7.run(JenaTxTdbProvider.java:1)
at 
com.ibm.team.jfs.rdf.internal.jena.tdb.JenaTxTdbProvider.storeOperation(JenaTxTdbProvider.java:208)
at 
com.ibm.team.jfs.rdf.internal.jena.tdb.JenaTxTdbProvider.readAndSetCurrentIndexSize(JenaTxTdbProvider.java:941)
at 
com.ibm.team.jfs.rdf.internal.jena.tdb.JenaTxTdbProvider.connect(JenaTxTdbProvider.java:488)
at 
com.ibm.team.jfs.rdf.internal.jena.JenaRdfService.connect(JenaRdfService.java:284)

> LeaveCriticalSection Error
> --
>
> Key: JENA-299
> URL: https://issues.apache.org/jira/browse/JENA-299
> Project: Apache Jena
>  Issue Type: Bug
>  Components: TDB
>Affects Versions: TDB 0.9.4
> Environment: windows 64 bit
>Reporter: Simon Helsen
>Assignee: Andy Seaborne
>Priority: Critical
> Fix For: TDB 0.9.4
>
> Attachments: LeaveCriticalSectionTestCase.zip
>
>
> I have attached a standalone test case, which, when run with the latest 
> snapshot produces the following exception:
> Exception in thread "main" com.hp.hpl.jena.shared.JenaException: 
> leaveCriticalSection: No lock held (main) Thread R/W: 0/0 :: Model R/W: 0/0 
> (thread: main)
>   at 
> com.hp.hpl.jena.shared.LockMRSW.leaveCriticalSection(LockMRSW.java:175)
>   at 
> com.hp.hpl.jena.tdb.transaction.TransactionManager$TSM_WriteBackEndTxn.readerFinishes(TransactionManager.java:210)
>   at 
> com.hp.hpl.jena.tdb.transaction.TransactionManager.readerFinishes(TransactionManager.java:723)
>   at 
> com.hp.hpl.jena.tdb.transaction.TransactionManager.noteTxnAbort(TransactionManager.java:587)
>   at 
> com.hp.hpl.jena.tdb.transaction.TransactionManager.notifyAbort(TransactionManager.java:445)
>   at 
> com.hp.hpl.jena.tdb.transaction.Transaction.abort(Transaction.java:162)
>   at 
> com.hp.hpl.jena.tdb.transaction.DatasetGraphTxn.abort(DatasetGraphTxn.java:45)
>   at 
> com.hp.hpl.jena.tdb.transaction.DatasetGraphTransaction._abort(DatasetGraphTransaction.java:156)
>   at 
> com.hp.hpl.jena.tdb.migrate.DatasetGraphTrackActive.abort(DatasetGraphTrackActive.java:68)
>   at com.hp.hpl.jena.sparql.core.DatasetImpl.abort(DatasetImpl.java:149)
>   at 
> com.ibm.jena.test.LeaveCriticalSectionErrorTest.storeOperation(LeaveCriticalSectionErrorTest.java:57)
>   at 
> com.ibm.jena.test.LeaveCriticalSectionErrorTest.query1(LeaveCriticalSectionErrorTest.java:105)
>   at 
> com.ibm.jena.test.LeaveCriticalSectionErrorTest.main(LeaveCriticalSectionErrorTest.java:156)
> The sequence in the test case is to run 2 queries, 1 write and then again 2 
> queries:
>   test.query1();
>   test.query1();
>   test.write1();
>   test.query1();
>   

[jira] [Updated] (JENA-301) RecordRangeIterator: records not strictly increasing

[Simon Helsen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/JENA-301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Simon Helsen updated JENA-301:
--

Attachment: PerformanceRegressionTest.java

> RecordRangeIterator: records not strictly increasing
> 
>
> Key: JENA-301
> URL: https://issues.apache.org/jira/browse/JENA-301
> Project: Apache Jena
>  Issue Type: Bug
>  Components: TDB
>Affects Versions: TDB 0.9.4
> Environment: windows 64 bit, oracle JRE 6
>Reporter: Simon Helsen
> Attachments: PerformanceRegressionTest.java
>
>
> When I tried to execute the provided test case for JENA-256, I ran into a TDB 
> Storage Exception. The scenario is like this
> 1) run the test application on a clean non-existing index. All should go 
> well, i.e. not exceptions, tests finish fine
> 2) then run the test application again on the same index. You'll end up with 
> the following stack trace at write operation 98:
> xception in thread "main" java.lang.RuntimeException: 
> com.hp.hpl.jena.tdb.base.StorageException: RecordRangeIterator: records not 
> strictly increasing: 
> 20ba002401490ad3 // 
> 08bc002401490271
>   at 
> com.ibm.jena.test.PerformanceRegressionTest.main(PerformanceRegressionTest.java:101)
> Caused by: com.hp.hpl.jena.tdb.base.StorageException: RecordRangeIterator: 
> records not strictly increasing: 
> 20ba002401490ad3 // 
> 08bc002401490271
>   at 
> com.hp.hpl.jena.tdb.base.recordbuffer.RecordRangeIterator.hasNext(RecordRangeIterator.java:124)
>   at org.openjena.atlas.iterator.Iter$4.hasNext(Iter.java:295)
>   at 
> com.hp.hpl.jena.tdb.sys.DatasetControlMRSW$IteratorCheckNotConcurrent.hasNext(DatasetControlMRSW.java:119)
>   at 
> com.hp.hpl.jena.tdb.graph.BulkUpdateHandlerTDB.removeWorker(BulkUpdateHandlerTDB.java:142)
>   at 
> com.hp.hpl.jena.tdb.graph.BulkUpdateHandlerTDB.removeAll(BulkUpdateHandlerTDB.java:102)
>   at com.hp.hpl.jena.rdf.model.impl.ModelCom.removeAll(ModelCom.java:372)
>   at 
> com.ibm.jena.test.PerformanceRegressionTest.main(PerformanceRegressionTest.java:78)
> The issue seems reproducible (I have seen it 3 times in a row when following 
> the scenario as described). Although I think this is exactly the same test 
> application as in JENA-256, I'll add it again for completeness

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (JENA-301) RecordRangeIterator: records not strictly increasing

[Simon Helsen (JIRA)]

Simon Helsen created JENA-301:
-

 Summary: RecordRangeIterator: records not strictly increasing
 Key: JENA-301
 URL: https://issues.apache.org/jira/browse/JENA-301
 Project: Apache Jena
  Issue Type: Bug
  Components: TDB
Affects Versions: TDB 0.9.4
 Environment: windows 64 bit, oracle JRE 6
Reporter: Simon Helsen
 Attachments: PerformanceRegressionTest.java

When I tried to execute the provided test case for JENA-256, I ran into a TDB 
Storage Exception. The scenario is like this

1) run the test application on a clean non-existing index. All should go well, 
i.e. not exceptions, tests finish fine
2) then run the test application again on the same index. You'll end up with 
the following stack trace at write operation 98:

xception in thread "main" java.lang.RuntimeException: 
com.hp.hpl.jena.tdb.base.StorageException: RecordRangeIterator: records not 
strictly increasing: 
20ba002401490ad3 // 
08bc002401490271
at 
com.ibm.jena.test.PerformanceRegressionTest.main(PerformanceRegressionTest.java:101)
Caused by: com.hp.hpl.jena.tdb.base.StorageException: RecordRangeIterator: 
records not strictly increasing: 
20ba002401490ad3 // 
08bc002401490271
at 
com.hp.hpl.jena.tdb.base.recordbuffer.RecordRangeIterator.hasNext(RecordRangeIterator.java:124)
at org.openjena.atlas.iterator.Iter$4.hasNext(Iter.java:295)
at 
com.hp.hpl.jena.tdb.sys.DatasetControlMRSW$IteratorCheckNotConcurrent.hasNext(DatasetControlMRSW.java:119)
at 
com.hp.hpl.jena.tdb.graph.BulkUpdateHandlerTDB.removeWorker(BulkUpdateHandlerTDB.java:142)
at 
com.hp.hpl.jena.tdb.graph.BulkUpdateHandlerTDB.removeAll(BulkUpdateHandlerTDB.java:102)
at com.hp.hpl.jena.rdf.model.impl.ModelCom.removeAll(ModelCom.java:372)
at 
com.ibm.jena.test.PerformanceRegressionTest.main(PerformanceRegressionTest.java:78)

The issue seems reproducible (I have seen it 3 times in a row when following 
the scenario as described). Although I think this is exactly the same test 
application as in JENA-256, I'll add it again for completeness

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-299) LeaveCriticalSection Error

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13437926#comment-13437926
 ] 

Simon Helsen commented on JENA-299:
---

Yes, sorry, when I said "testing things out now", I meant, I will use the 
latest snapshot (build from the 20th), but I will also follow the 
recommendations

> LeaveCriticalSection Error
> --
>
> Key: JENA-299
> URL: https://issues.apache.org/jira/browse/JENA-299
> Project: Apache Jena
>  Issue Type: Bug
>  Components: TDB
>Affects Versions: TDB 0.9.4
> Environment: windows 64 bit
>Reporter: Simon Helsen
>Assignee: Andy Seaborne
>Priority: Critical
> Fix For: TDB 0.9.4
>
> Attachments: LeaveCriticalSectionTestCase.zip
>
>
> I have attached a standalone test case, which, when run with the latest 
> snapshot produces the following exception:
> Exception in thread "main" com.hp.hpl.jena.shared.JenaException: 
> leaveCriticalSection: No lock held (main) Thread R/W: 0/0 :: Model R/W: 0/0 
> (thread: main)
>   at 
> com.hp.hpl.jena.shared.LockMRSW.leaveCriticalSection(LockMRSW.java:175)
>   at 
> com.hp.hpl.jena.tdb.transaction.TransactionManager$TSM_WriteBackEndTxn.readerFinishes(TransactionManager.java:210)
>   at 
> com.hp.hpl.jena.tdb.transaction.TransactionManager.readerFinishes(TransactionManager.java:723)
>   at 
> com.hp.hpl.jena.tdb.transaction.TransactionManager.noteTxnAbort(TransactionManager.java:587)
>   at 
> com.hp.hpl.jena.tdb.transaction.TransactionManager.notifyAbort(TransactionManager.java:445)
>   at 
> com.hp.hpl.jena.tdb.transaction.Transaction.abort(Transaction.java:162)
>   at 
> com.hp.hpl.jena.tdb.transaction.DatasetGraphTxn.abort(DatasetGraphTxn.java:45)
>   at 
> com.hp.hpl.jena.tdb.transaction.DatasetGraphTransaction._abort(DatasetGraphTransaction.java:156)
>   at 
> com.hp.hpl.jena.tdb.migrate.DatasetGraphTrackActive.abort(DatasetGraphTrackActive.java:68)
>   at com.hp.hpl.jena.sparql.core.DatasetImpl.abort(DatasetImpl.java:149)
>   at 
> com.ibm.jena.test.LeaveCriticalSectionErrorTest.storeOperation(LeaveCriticalSectionErrorTest.java:57)
>   at 
> com.ibm.jena.test.LeaveCriticalSectionErrorTest.query1(LeaveCriticalSectionErrorTest.java:105)
>   at 
> com.ibm.jena.test.LeaveCriticalSectionErrorTest.main(LeaveCriticalSectionErrorTest.java:156)
> The sequence in the test case is to run 2 queries, 1 write and then again 2 
> queries:
>   test.query1();
>   test.query1();
>   test.write1();
>   test.query1();
>   test.query1();
> Somehow, the following sequence did not produce the exception:
>   test.query1();
>   test.query1();
>   test.write1();
>   test.query1();
> Note that the test case does not check the correctness of any results.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-299) LeaveCriticalSection Error

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13437918#comment-13437918
 ] 

Simon Helsen commented on JENA-299:
---

thanks Andy. I am adjusting my code with your recommendations. Testing things 
out now

> LeaveCriticalSection Error
> --
>
> Key: JENA-299
> URL: https://issues.apache.org/jira/browse/JENA-299
> Project: Apache Jena
>  Issue Type: Bug
>  Components: TDB
>Affects Versions: TDB 0.9.4
> Environment: windows 64 bit
>Reporter: Simon Helsen
>Assignee: Andy Seaborne
>Priority: Critical
> Fix For: TDB 0.9.4
>
> Attachments: LeaveCriticalSectionTestCase.zip
>
>
> I have attached a standalone test case, which, when run with the latest 
> snapshot produces the following exception:
> Exception in thread "main" com.hp.hpl.jena.shared.JenaException: 
> leaveCriticalSection: No lock held (main) Thread R/W: 0/0 :: Model R/W: 0/0 
> (thread: main)
>   at 
> com.hp.hpl.jena.shared.LockMRSW.leaveCriticalSection(LockMRSW.java:175)
>   at 
> com.hp.hpl.jena.tdb.transaction.TransactionManager$TSM_WriteBackEndTxn.readerFinishes(TransactionManager.java:210)
>   at 
> com.hp.hpl.jena.tdb.transaction.TransactionManager.readerFinishes(TransactionManager.java:723)
>   at 
> com.hp.hpl.jena.tdb.transaction.TransactionManager.noteTxnAbort(TransactionManager.java:587)
>   at 
> com.hp.hpl.jena.tdb.transaction.TransactionManager.notifyAbort(TransactionManager.java:445)
>   at 
> com.hp.hpl.jena.tdb.transaction.Transaction.abort(Transaction.java:162)
>   at 
> com.hp.hpl.jena.tdb.transaction.DatasetGraphTxn.abort(DatasetGraphTxn.java:45)
>   at 
> com.hp.hpl.jena.tdb.transaction.DatasetGraphTransaction._abort(DatasetGraphTransaction.java:156)
>   at 
> com.hp.hpl.jena.tdb.migrate.DatasetGraphTrackActive.abort(DatasetGraphTrackActive.java:68)
>   at com.hp.hpl.jena.sparql.core.DatasetImpl.abort(DatasetImpl.java:149)
>   at 
> com.ibm.jena.test.LeaveCriticalSectionErrorTest.storeOperation(LeaveCriticalSectionErrorTest.java:57)
>   at 
> com.ibm.jena.test.LeaveCriticalSectionErrorTest.query1(LeaveCriticalSectionErrorTest.java:105)
>   at 
> com.ibm.jena.test.LeaveCriticalSectionErrorTest.main(LeaveCriticalSectionErrorTest.java:156)
> The sequence in the test case is to run 2 queries, 1 write and then again 2 
> queries:
>   test.query1();
>   test.query1();
>   test.write1();
>   test.query1();
>   test.query1();
> Somehow, the following sequence did not produce the exception:
>   test.query1();
>   test.query1();
>   test.write1();
>   test.query1();
> Note that the test case does not check the correctness of any results.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-299) LeaveCriticalSection Error

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13437011#comment-13437011
 ] 

Simon Helsen commented on JENA-299:
---

as an aside but related: I wonder if the transactional code would benefit from 
some additional optional trace logging. E.g. as a client I have no access to a 
transaction id. Yesterday, I wanted to actually write some tracing code to 
figure out if I accidentally was sharing a transaction over multiple thread, 
but I was unable to get the id of a given transaction. But this invariant is 
imposed by TDB/Tx, so perhaps the Tx code could trace some of this (checking 
would in theory be possible but comes at a performance penalty which should be 
avoid IMO) 

> LeaveCriticalSection Error
> --
>
> Key: JENA-299
> URL: https://issues.apache.org/jira/browse/JENA-299
> Project: Apache Jena
>  Issue Type: Bug
>  Components: TDB
>Affects Versions: TDB 0.9.4
> Environment: windows 64 bit
>Reporter: Simon Helsen
>Priority: Critical
> Attachments: LeaveCriticalSectionTestCase.zip
>
>
> I have attached a standalone test case, which, when run with the latest 
> snapshot produces the following exception:
> Exception in thread "main" com.hp.hpl.jena.shared.JenaException: 
> leaveCriticalSection: No lock held (main) Thread R/W: 0/0 :: Model R/W: 0/0 
> (thread: main)
>   at 
> com.hp.hpl.jena.shared.LockMRSW.leaveCriticalSection(LockMRSW.java:175)
>   at 
> com.hp.hpl.jena.tdb.transaction.TransactionManager$TSM_WriteBackEndTxn.readerFinishes(TransactionManager.java:210)
>   at 
> com.hp.hpl.jena.tdb.transaction.TransactionManager.readerFinishes(TransactionManager.java:723)
>   at 
> com.hp.hpl.jena.tdb.transaction.TransactionManager.noteTxnAbort(TransactionManager.java:587)
>   at 
> com.hp.hpl.jena.tdb.transaction.TransactionManager.notifyAbort(TransactionManager.java:445)
>   at 
> com.hp.hpl.jena.tdb.transaction.Transaction.abort(Transaction.java:162)
>   at 
> com.hp.hpl.jena.tdb.transaction.DatasetGraphTxn.abort(DatasetGraphTxn.java:45)
>   at 
> com.hp.hpl.jena.tdb.transaction.DatasetGraphTransaction._abort(DatasetGraphTransaction.java:156)
>   at 
> com.hp.hpl.jena.tdb.migrate.DatasetGraphTrackActive.abort(DatasetGraphTrackActive.java:68)
>   at com.hp.hpl.jena.sparql.core.DatasetImpl.abort(DatasetImpl.java:149)
>   at 
> com.ibm.jena.test.LeaveCriticalSectionErrorTest.storeOperation(LeaveCriticalSectionErrorTest.java:57)
>   at 
> com.ibm.jena.test.LeaveCriticalSectionErrorTest.query1(LeaveCriticalSectionErrorTest.java:105)
>   at 
> com.ibm.jena.test.LeaveCriticalSectionErrorTest.main(LeaveCriticalSectionErrorTest.java:156)
> The sequence in the test case is to run 2 queries, 1 write and then again 2 
> queries:
>   test.query1();
>   test.query1();
>   test.write1();
>   test.query1();
>   test.query1();
> Somehow, the following sequence did not produce the exception:
>   test.query1();
>   test.query1();
>   test.write1();
>   test.query1();
> Note that the test case does not check the correctness of any results.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-299) LeaveCriticalSection Error

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13436940#comment-13436940
 ] 

Simon Helsen commented on JENA-299:
---

yes, I think in my isolted test, the logging was not connected. I had seen 
these in the suite in our product. What does this mean btw? What is a 
non-active transaction? 

> LeaveCriticalSection Error
> --
>
> Key: JENA-299
> URL: https://issues.apache.org/jira/browse/JENA-299
> Project: Apache Jena
>  Issue Type: Bug
>  Components: TDB
>Affects Versions: TDB 0.9.4
> Environment: windows 64 bit
>Reporter: Simon Helsen
>Priority: Critical
> Attachments: LeaveCriticalSectionTestCase.zip
>
>
> I have attached a standalone test case, which, when run with the latest 
> snapshot produces the following exception:
> Exception in thread "main" com.hp.hpl.jena.shared.JenaException: 
> leaveCriticalSection: No lock held (main) Thread R/W: 0/0 :: Model R/W: 0/0 
> (thread: main)
>   at 
> com.hp.hpl.jena.shared.LockMRSW.leaveCriticalSection(LockMRSW.java:175)
>   at 
> com.hp.hpl.jena.tdb.transaction.TransactionManager$TSM_WriteBackEndTxn.readerFinishes(TransactionManager.java:210)
>   at 
> com.hp.hpl.jena.tdb.transaction.TransactionManager.readerFinishes(TransactionManager.java:723)
>   at 
> com.hp.hpl.jena.tdb.transaction.TransactionManager.noteTxnAbort(TransactionManager.java:587)
>   at 
> com.hp.hpl.jena.tdb.transaction.TransactionManager.notifyAbort(TransactionManager.java:445)
>   at 
> com.hp.hpl.jena.tdb.transaction.Transaction.abort(Transaction.java:162)
>   at 
> com.hp.hpl.jena.tdb.transaction.DatasetGraphTxn.abort(DatasetGraphTxn.java:45)
>   at 
> com.hp.hpl.jena.tdb.transaction.DatasetGraphTransaction._abort(DatasetGraphTransaction.java:156)
>   at 
> com.hp.hpl.jena.tdb.migrate.DatasetGraphTrackActive.abort(DatasetGraphTrackActive.java:68)
>   at com.hp.hpl.jena.sparql.core.DatasetImpl.abort(DatasetImpl.java:149)
>   at 
> com.ibm.jena.test.LeaveCriticalSectionErrorTest.storeOperation(LeaveCriticalSectionErrorTest.java:57)
>   at 
> com.ibm.jena.test.LeaveCriticalSectionErrorTest.query1(LeaveCriticalSectionErrorTest.java:105)
>   at 
> com.ibm.jena.test.LeaveCriticalSectionErrorTest.main(LeaveCriticalSectionErrorTest.java:156)
> The sequence in the test case is to run 2 queries, 1 write and then again 2 
> queries:
>   test.query1();
>   test.query1();
>   test.write1();
>   test.query1();
>   test.query1();
> Somehow, the following sequence did not produce the exception:
>   test.query1();
>   test.query1();
>   test.write1();
>   test.query1();
> Note that the test case does not check the correctness of any results.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (JENA-299) LeaveCriticalSection Error

[Simon Helsen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/JENA-299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Simon Helsen updated JENA-299:
--

Attachment: LeaveCriticalSectionTestCase.zip

This is the whole test project. Just adjust the classpath to the HEAD of 
ARQ/TDB/CORE/IRI or move the test out

> LeaveCriticalSection Error
> --
>
> Key: JENA-299
> URL: https://issues.apache.org/jira/browse/JENA-299
> Project: Apache Jena
>  Issue Type: Bug
>  Components: TDB
>Affects Versions: TDB 0.9.4
> Environment: windows 64 bit
>Reporter: Simon Helsen
>Priority: Critical
> Attachments: LeaveCriticalSectionTestCase.zip
>
>
> I have attached a standalone test case, which, when run with the latest 
> snapshot produces the following exception:
> Exception in thread "main" com.hp.hpl.jena.shared.JenaException: 
> leaveCriticalSection: No lock held (main) Thread R/W: 0/0 :: Model R/W: 0/0 
> (thread: main)
>   at 
> com.hp.hpl.jena.shared.LockMRSW.leaveCriticalSection(LockMRSW.java:175)
>   at 
> com.hp.hpl.jena.tdb.transaction.TransactionManager$TSM_WriteBackEndTxn.readerFinishes(TransactionManager.java:210)
>   at 
> com.hp.hpl.jena.tdb.transaction.TransactionManager.readerFinishes(TransactionManager.java:723)
>   at 
> com.hp.hpl.jena.tdb.transaction.TransactionManager.noteTxnAbort(TransactionManager.java:587)
>   at 
> com.hp.hpl.jena.tdb.transaction.TransactionManager.notifyAbort(TransactionManager.java:445)
>   at 
> com.hp.hpl.jena.tdb.transaction.Transaction.abort(Transaction.java:162)
>   at 
> com.hp.hpl.jena.tdb.transaction.DatasetGraphTxn.abort(DatasetGraphTxn.java:45)
>   at 
> com.hp.hpl.jena.tdb.transaction.DatasetGraphTransaction._abort(DatasetGraphTransaction.java:156)
>   at 
> com.hp.hpl.jena.tdb.migrate.DatasetGraphTrackActive.abort(DatasetGraphTrackActive.java:68)
>   at com.hp.hpl.jena.sparql.core.DatasetImpl.abort(DatasetImpl.java:149)
>   at 
> com.ibm.jena.test.LeaveCriticalSectionErrorTest.storeOperation(LeaveCriticalSectionErrorTest.java:57)
>   at 
> com.ibm.jena.test.LeaveCriticalSectionErrorTest.query1(LeaveCriticalSectionErrorTest.java:105)
>   at 
> com.ibm.jena.test.LeaveCriticalSectionErrorTest.main(LeaveCriticalSectionErrorTest.java:156)
> The sequence in the test case is to run 2 queries, 1 write and then again 2 
> queries:
>   test.query1();
>   test.query1();
>   test.write1();
>   test.query1();
>   test.query1();
> Somehow, the following sequence did not produce the exception:
>   test.query1();
>   test.query1();
>   test.write1();
>   test.query1();
> Note that the test case does not check the correctness of any results.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (JENA-299) LeaveCriticalSection Error

[Simon Helsen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/JENA-299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Simon Helsen updated JENA-299:
--

Affects Version/s: (was: TDB 0.9.3)

> LeaveCriticalSection Error
> --
>
> Key: JENA-299
> URL: https://issues.apache.org/jira/browse/JENA-299
> Project: Apache Jena
>  Issue Type: Bug
>  Components: TDB
>Affects Versions: TDB 0.9.4
> Environment: windows 64 bit
>Reporter: Simon Helsen
>Priority: Critical
> Attachments: LeaveCriticalSectionTestCase.zip
>
>
> I have attached a standalone test case, which, when run with the latest 
> snapshot produces the following exception:
> Exception in thread "main" com.hp.hpl.jena.shared.JenaException: 
> leaveCriticalSection: No lock held (main) Thread R/W: 0/0 :: Model R/W: 0/0 
> (thread: main)
>   at 
> com.hp.hpl.jena.shared.LockMRSW.leaveCriticalSection(LockMRSW.java:175)
>   at 
> com.hp.hpl.jena.tdb.transaction.TransactionManager$TSM_WriteBackEndTxn.readerFinishes(TransactionManager.java:210)
>   at 
> com.hp.hpl.jena.tdb.transaction.TransactionManager.readerFinishes(TransactionManager.java:723)
>   at 
> com.hp.hpl.jena.tdb.transaction.TransactionManager.noteTxnAbort(TransactionManager.java:587)
>   at 
> com.hp.hpl.jena.tdb.transaction.TransactionManager.notifyAbort(TransactionManager.java:445)
>   at 
> com.hp.hpl.jena.tdb.transaction.Transaction.abort(Transaction.java:162)
>   at 
> com.hp.hpl.jena.tdb.transaction.DatasetGraphTxn.abort(DatasetGraphTxn.java:45)
>   at 
> com.hp.hpl.jena.tdb.transaction.DatasetGraphTransaction._abort(DatasetGraphTransaction.java:156)
>   at 
> com.hp.hpl.jena.tdb.migrate.DatasetGraphTrackActive.abort(DatasetGraphTrackActive.java:68)
>   at com.hp.hpl.jena.sparql.core.DatasetImpl.abort(DatasetImpl.java:149)
>   at 
> com.ibm.jena.test.LeaveCriticalSectionErrorTest.storeOperation(LeaveCriticalSectionErrorTest.java:57)
>   at 
> com.ibm.jena.test.LeaveCriticalSectionErrorTest.query1(LeaveCriticalSectionErrorTest.java:105)
>   at 
> com.ibm.jena.test.LeaveCriticalSectionErrorTest.main(LeaveCriticalSectionErrorTest.java:156)
> The sequence in the test case is to run 2 queries, 1 write and then again 2 
> queries:
>   test.query1();
>   test.query1();
>   test.write1();
>   test.query1();
>   test.query1();
> Somehow, the following sequence did not produce the exception:
>   test.query1();
>   test.query1();
>   test.write1();
>   test.query1();
> Note that the test case does not check the correctness of any results.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (JENA-299) LeaveCriticalSection Error

[Simon Helsen (JIRA)]

Simon Helsen created JENA-299:
-

 Summary: LeaveCriticalSection Error
 Key: JENA-299
 URL: https://issues.apache.org/jira/browse/JENA-299
 Project: Apache Jena
  Issue Type: Bug
  Components: TDB
Affects Versions: TDB 0.9.3, TDB 0.9.4
 Environment: windows 64 bit
Reporter: Simon Helsen
Priority: Critical


I have attached a standalone test case, which, when run with the latest 
snapshot produces the following exception:

Exception in thread "main" com.hp.hpl.jena.shared.JenaException: 
leaveCriticalSection: No lock held (main) Thread R/W: 0/0 :: Model R/W: 0/0 
(thread: main)
at 
com.hp.hpl.jena.shared.LockMRSW.leaveCriticalSection(LockMRSW.java:175)
at 
com.hp.hpl.jena.tdb.transaction.TransactionManager$TSM_WriteBackEndTxn.readerFinishes(TransactionManager.java:210)
at 
com.hp.hpl.jena.tdb.transaction.TransactionManager.readerFinishes(TransactionManager.java:723)
at 
com.hp.hpl.jena.tdb.transaction.TransactionManager.noteTxnAbort(TransactionManager.java:587)
at 
com.hp.hpl.jena.tdb.transaction.TransactionManager.notifyAbort(TransactionManager.java:445)
at 
com.hp.hpl.jena.tdb.transaction.Transaction.abort(Transaction.java:162)
at 
com.hp.hpl.jena.tdb.transaction.DatasetGraphTxn.abort(DatasetGraphTxn.java:45)
at 
com.hp.hpl.jena.tdb.transaction.DatasetGraphTransaction._abort(DatasetGraphTransaction.java:156)
at 
com.hp.hpl.jena.tdb.migrate.DatasetGraphTrackActive.abort(DatasetGraphTrackActive.java:68)
at com.hp.hpl.jena.sparql.core.DatasetImpl.abort(DatasetImpl.java:149)
at 
com.ibm.jena.test.LeaveCriticalSectionErrorTest.storeOperation(LeaveCriticalSectionErrorTest.java:57)
at 
com.ibm.jena.test.LeaveCriticalSectionErrorTest.query1(LeaveCriticalSectionErrorTest.java:105)
at 
com.ibm.jena.test.LeaveCriticalSectionErrorTest.main(LeaveCriticalSectionErrorTest.java:156)

The sequence in the test case is to run 2 queries, 1 write and then again 2 
queries:

test.query1();
test.query1();
test.write1();
test.query1();
test.query1();

Somehow, the following sequence did not produce the exception:

test.query1();
test.query1();
test.write1();
test.query1();

Note that the test case does not check the correctness of any results.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-294) TransformFilterEquality does not handle starting OPTIONAL well

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13435959#comment-13435959
 ] 

Simon Helsen commented on JENA-294:
---

Andy, I am confused by you answer "yes, it does". Are you saying that the 
proposed patch works correctly for that case? And if so, are you saying that 

(prefix ((: ;))
  (assign ((?x :x))
(conditional
  (conditional
(table unit)
(bgp (triple :x :r ?s)))
  (bgp (triple :x :q ?o) 

is a correct plan?

> TransformFilterEquality does not handle starting OPTIONAL well
> --
>
> Key: JENA-294
> URL: https://issues.apache.org/jira/browse/JENA-294
> Project: Apache Jena
>  Issue Type: Improvement
>  Components: ARQ
>Affects Versions: Jena 2.7.4
>Reporter: Simon Helsen
> Attachments: jena-294-afs-1.patch, patch.txt, patch.txt, 
> test-filter-equality.zip
>
>
> There was one other case where our tests were stuck on a very slow query 
> execution because transformFilterEquality failed to optimize. The problem is 
> that the optimizer gives up whenever the WHERE clause starts with an OPTIONAL 
> clause. The reason is that the generated algebraic formula starts with a 
> TableUnit and this is not handled correctly. 
> I have attached a patch which fixes the problem

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-294) TransformFilterEquality does not handle starting OPTIONAL well

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13435436#comment-13435436
 ] 

Simon Helsen commented on JENA-294:
---

When I apply your patch to the trunk and try it on the following query:

PREFIX :  
SELECT *
{
   OPTIONAL { ?x :r ?s }
   OPTIONAL { ?x :q ?o }
   FILTER(?x = :x)
} 

I get the following plan:

(prefix ((: ))
  (assign ((?x :x))
(conditional
  (conditional
(table unit)
(bgp (triple :x :r ?s)))
  (bgp (triple :x :q ?o)

Does this not exhibit the same problem as discussed earlier? I.e. the bindings 
on ?x for :r and :q may not exist, yet a row would be returned with a binding 
for :x

> TransformFilterEquality does not handle starting OPTIONAL well
> --
>
> Key: JENA-294
> URL: https://issues.apache.org/jira/browse/JENA-294
> Project: Apache Jena
>  Issue Type: Improvement
>  Components: ARQ
>Affects Versions: Jena 2.7.4
>Reporter: Simon Helsen
> Attachments: jena-294-afs-1.patch, patch.txt, patch.txt, 
> test-filter-equality.zip
>
>
> There was one other case where our tests were stuck on a very slow query 
> execution because transformFilterEquality failed to optimize. The problem is 
> that the optimizer gives up whenever the WHERE clause starts with an OPTIONAL 
> clause. The reason is that the generated algebraic formula starts with a 
> TableUnit and this is not handled correctly. 
> I have attached a patch which fixes the problem

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-294) TransformFilterEquality does not handle starting OPTIONAL well

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13435237#comment-13435237
 ] 

Simon Helsen commented on JENA-294:
---

thanks Andy. I'll take a look at it and also test it locally within our 
framework, even though those queries are more conservative 

> TransformFilterEquality does not handle starting OPTIONAL well
> --
>
> Key: JENA-294
> URL: https://issues.apache.org/jira/browse/JENA-294
> Project: Apache Jena
>  Issue Type: Improvement
>  Components: ARQ
>Affects Versions: Jena 2.7.4
>Reporter: Simon Helsen
> Attachments: jena-294-afs-1.patch, patch.txt, patch.txt, 
> test-filter-equality.zip
>
>
> There was one other case where our tests were stuck on a very slow query 
> execution because transformFilterEquality failed to optimize. The problem is 
> that the optimizer gives up whenever the WHERE clause starts with an OPTIONAL 
> clause. The reason is that the generated algebraic formula starts with a 
> TableUnit and this is not handled correctly. 
> I have attached a patch which fixes the problem

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (JENA-294) TransformFilterEquality does not handle starting OPTIONAL well

[Simon Helsen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/JENA-294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Simon Helsen updated JENA-294:
--

Attachment: patch.txt

> TransformFilterEquality does not handle starting OPTIONAL well
> --
>
> Key: JENA-294
> URL: https://issues.apache.org/jira/browse/JENA-294
> Project: Apache Jena
>  Issue Type: Improvement
>  Components: ARQ
>Affects Versions: Jena 2.7.4
>Reporter: Simon Helsen
> Attachments: patch.txt, patch.txt
>
>
> There was one other case where our tests were stuck on a very slow query 
> execution because transformFilterEquality failed to optimize. The problem is 
> that the optimizer gives up whenever the WHERE clause starts with an OPTIONAL 
> clause. The reason is that the generated algebraic formula starts with a 
> TableUnit and this is not handled correctly. 
> I have attached a patch which fixes the problem

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-294) TransformFilterEquality does not handle starting OPTIONAL well

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434589#comment-13434589
 ] 

Simon Helsen commented on JENA-294:
---

ah, I see now, it seems that the expansion (substitute) should take place 
regardless, but the assignment should only happen if the conditional evaluates 
to something. So it should be pushed inside and I see now that this is what you 
were saying in one of the comments above. 

Btw, the reason our example does not fail is that the WHERE clause also has a 
non-optional condition on the filter variable, so even if the optional does not 
match, the assignment is always correct. 

I am attaching a new patch. If an assignment sits on a conditional or leftJoin, 
it shifts it inside the RHS. Now, I must say I am not entirely sure how nested 
conditionals are evaluated, so perhaps this is not quite right. E.g. I am 
wondering if the assignment should also be pushed into the LHS and then 
recursively down. The substitution itself is not affected I'd say and the 
precondition check remains the same as well because a LHS in these left joins 
which is a unit does not.

So, e.g. query

PREFIX : 

SELECT *
{
  OPTIONAL { ?x :p ?z }
  OPTIONAL { ?y :q ?z }
  FILTER(?z = :a)
} 

turns into

(prefix ((: ))
  (conditional
(conditional
  (table unit)
  (bgp (triple ?x :p :a)))
(assign ((?z :a))
  (bgp (triple ?y :q :a)

but query

PREFIX : 

SELECT *
{
  OPTIONAL { ?x :p ?z }
  OPTIONAL { ?y :q ?z }
  ?w :r ?z
  FILTER(?z = :a)
} 

(which is more like our use case right now) produces this plan:

(prefix ((: ))
  (assign ((?z :a))
(sequence
  (conditional
(conditional
  (table unit)
  (bgp (triple ?x :p :a)))
(bgp (triple ?y :q :a)))
  (bgp (triple ?w :r :a)

and your example 

PREFIX : 

SELECT *
{
   OPTIONAL { ?x :q ?o }
   FILTER(?x = :x)
} 

produces plan

(prefix ((: ))
  (conditional
(table unit)
(assign ((?x :x))
  (bgp (triple :x :q ?o)

> TransformFilterEquality does not handle starting OPTIONAL well
> --
>
> Key: JENA-294
> URL: https://issues.apache.org/jira/browse/JENA-294
> Project: Apache Jena
>  Issue Type: Improvement
>  Components: ARQ
>Affects Versions: Jena 2.7.4
>Reporter: Simon Helsen
> Attachments: patch.txt
>
>
> There was one other case where our tests were stuck on a very slow query 
> execution because transformFilterEquality failed to optimize. The problem is 
> that the optimizer gives up whenever the WHERE clause starts with an OPTIONAL 
> clause. The reason is that the generated algebraic formula starts with a 
> TableUnit and this is not handled correctly. 
> I have attached a patch which fixes the problem

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-294) TransformFilterEquality does not handle starting OPTIONAL well

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434398#comment-13434398
 ] 

Simon Helsen commented on JENA-294:
---

right, but because we are under time pressure (internally) I need at least a 
probable correct solution for the simple case which was affecting us. I am not 
quite following your statement "It may not be necessary, or it may be simpler, 
after leftjoin to condition as the precondition for that rewrite depends on 
scoping of variables as well"

> TransformFilterEquality does not handle starting OPTIONAL well
> --
>
> Key: JENA-294
> URL: https://issues.apache.org/jira/browse/JENA-294
> Project: Apache Jena
>  Issue Type: Improvement
>  Components: ARQ
>Affects Versions: Jena 2.7.4
>Reporter: Simon Helsen
> Attachments: patch.txt
>
>
> There was one other case where our tests were stuck on a very slow query 
> execution because transformFilterEquality failed to optimize. The problem is 
> that the optimizer gives up whenever the WHERE clause starts with an OPTIONAL 
> clause. The reason is that the generated algebraic formula starts with a 
> TableUnit and this is not handled correctly. 
> I have attached a patch which fixes the problem

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-294) TransformFilterEquality does not handle starting OPTIONAL well

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434381#comment-13434381
 ] 

Simon Helsen commented on JENA-294:
---

how about changing the check around line 132 to the following:

 if (isUnitTable(opLeft)) {
opLeft = opleftjoin.getRight();
 }

I guess this is what you suggest in your comment above, except for more general 
cases than just the unit left hand side. 

> TransformFilterEquality does not handle starting OPTIONAL well
> --
>
> Key: JENA-294
> URL: https://issues.apache.org/jira/browse/JENA-294
> Project: Apache Jena
>  Issue Type: Improvement
>  Components: ARQ
>Affects Versions: Jena 2.7.4
>Reporter: Simon Helsen
> Attachments: patch.txt
>
>
> There was one other case where our tests were stuck on a very slow query 
> execution because transformFilterEquality failed to optimize. The problem is 
> that the optimizer gives up whenever the WHERE clause starts with an OPTIONAL 
> clause. The reason is that the generated algebraic formula starts with a 
> TableUnit and this is not handled correctly. 
> I have attached a patch which fixes the problem

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (JENA-294) TransformFilterEquality does not handle starting OPTIONAL well

[Simon Helsen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/JENA-294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Simon Helsen updated JENA-294:
--

Attachment: patch.txt

> TransformFilterEquality does not handle starting OPTIONAL well
> --
>
> Key: JENA-294
> URL: https://issues.apache.org/jira/browse/JENA-294
> Project: Apache Jena
>  Issue Type: Bug
>  Components: ARQ
>Affects Versions: Jena 2.7.4
>Reporter: Simon Helsen
> Attachments: patch.txt
>
>
> There was one other case where our tests were stuck on a very slow query 
> execution because transformFilterEquality failed to optimize. The problem is 
> that the optimizer gives up whenever the WHERE clause starts with an OPTIONAL 
> clause. The reason is that the generated algebraic formula starts with a 
> TableUnit and this is not handled correctly. 
> I have attached a patch which fixes the problem

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-294) TransformFilterEquality does not handle starting OPTIONAL well

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13433427#comment-13433427
 ] 

Simon Helsen commented on JENA-294:
---

this is related to JENA 283

> TransformFilterEquality does not handle starting OPTIONAL well
> --
>
> Key: JENA-294
> URL: https://issues.apache.org/jira/browse/JENA-294
> Project: Apache Jena
>  Issue Type: Bug
>  Components: ARQ
>Affects Versions: Jena 2.7.4
>Reporter: Simon Helsen
> Attachments: patch.txt
>
>
> There was one other case where our tests were stuck on a very slow query 
> execution because transformFilterEquality failed to optimize. The problem is 
> that the optimizer gives up whenever the WHERE clause starts with an OPTIONAL 
> clause. The reason is that the generated algebraic formula starts with a 
> TableUnit and this is not handled correctly. 
> I have attached a patch which fixes the problem

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (JENA-294) TransformFilterEquality does not handle starting OPTIONAL well

[Simon Helsen (JIRA)]

Simon Helsen created JENA-294:
-

 Summary: TransformFilterEquality does not handle starting OPTIONAL 
well
 Key: JENA-294
 URL: https://issues.apache.org/jira/browse/JENA-294
 Project: Apache Jena
  Issue Type: Bug
  Components: ARQ
Affects Versions: Jena 2.7.4
Reporter: Simon Helsen


There was one other case where our tests were stuck on a very slow query 
execution because transformFilterEquality failed to optimize. The problem is 
that the optimizer gives up whenever the WHERE clause starts with an OPTIONAL 
clause. The reason is that the generated algebraic formula starts with a 
TableUnit and this is not handled correctly. 

I have attached a patch which fixes the problem

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-256) Significant performance regression (TDB?) on 2.7.1 RC compared to May 15 build

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13418609#comment-13418609
 ] 

Simon Helsen commented on JENA-256:
---

ok, so I tried the drop, but that didn't make a difference. Then I realized, 
looking at the code that I was probably running in mapped mode which, on 
Windows is shaky (and known to be slower), so I explicitly changed to use 
direct mode. The first run went fine (and fast), but the second run failed with 
the following exception (systematically):

Exception in thread "main" com.hp.hpl.jena.tdb.base.block.BlockException: 
BlockAccessBase: Bounds exception: 
C:\Temp\jenaTests\indexTestLocation\GSPO.dat: (40,40)
at 
com.hp.hpl.jena.tdb.base.file.BlockAccessBase.check(BlockAccessBase.java:107)
at 
com.hp.hpl.jena.tdb.base.file.BlockAccessBase.check(BlockAccessBase.java:114)
at 
com.hp.hpl.jena.tdb.base.file.BlockAccessDirect.write(BlockAccessDirect.java:85)
at 
com.hp.hpl.jena.tdb.base.file.BlockAccessDirect.overwrite(BlockAccessDirect.java:104)
at 
com.hp.hpl.jena.tdb.base.block.BlockMgrFileAccess.overwrite(BlockMgrFileAccess.java:102)
at 
com.hp.hpl.jena.tdb.base.block.BlockMgrWrapper.overwrite(BlockMgrWrapper.java:89)
at 
com.hp.hpl.jena.tdb.base.block.BlockMgrSync.overwrite(BlockMgrSync.java:90)
at 
com.hp.hpl.jena.tdb.base.block.BlockMgrCache.overwrite(BlockMgrCache.java:206)
at 
com.hp.hpl.jena.tdb.transaction.JournalControl.replay(JournalControl.java:305)
at 
com.hp.hpl.jena.tdb.transaction.JournalControl.recoverSegment(JournalControl.java:175)
at 
com.hp.hpl.jena.tdb.transaction.JournalControl.recoverFromJournal(JournalControl.java:128)
at 
com.hp.hpl.jena.tdb.StoreConnection._makeAndCache(StoreConnection.java:237)
at com.hp.hpl.jena.tdb.StoreConnection.make(StoreConnection.java:196)
at 
com.hp.hpl.jena.tdb.transaction.DatasetGraphTransaction.(DatasetGraphTransaction.java:76)
at com.hp.hpl.jena.tdb.sys.TDBMakerTxn._create(TDBMakerTxn.java:49)
at 
com.hp.hpl.jena.tdb.sys.TDBMakerTxn.createDatasetGraph(TDBMakerTxn.java:37)
at 
com.hp.hpl.jena.tdb.TDBFactory._createDatasetGraph(TDBFactory.java:104)
at com.hp.hpl.jena.tdb.TDBFactory.createDatasetGraph(TDBFactory.java:73)
at com.hp.hpl.jena.tdb.TDBFactory.createDataset(TDBFactory.java:52)
at com.hp.hpl.jena.tdb.TDBFactory.createDataset(TDBFactory.java:48)
at 
com.ibm.jena.test.PerformanceRegressionTest.main(PerformanceRegressionTest.java:54)

It is reproducible and the only thing I did was add the following static block:

static {
ARQ.getContext().set(SystemTDB.symFileMode, "direct");
}

Can you reproduce this?


> Significant performance regression (TDB?) on 2.7.1 RC compared to May 15 build
> --
>
> Key: JENA-256
> URL: https://issues.apache.org/jira/browse/JENA-256
> Project: Apache Jena
>  Issue Type: Bug
>  Components: TDB
>Affects Versions: Jena 2.7.1
> Environment: Windows 7, 64 bit, tested against 2.7.1 RC from June 9 
> versus a build from May 15
>Reporter: Simon Helsen
>Assignee: Andy Seaborne
>Priority: Critical
> Fix For: TDB 0.9.3
>
> Attachments: 271May15Output1.txt, 271May15Output2.txt, 
> 271May15Output3.txt, 271May15Output4.txt, 271RCOutput1.txt, 271RCOutput2.txt, 
> 271RCOutput3.txt, 271RCOutput4.txt, PerformanceRegressionTest.java
>
>
> See also 
> http://mail-archives.apache.org/mod_mbox/jena-dev/201206.mbox/%3C4FD9F19E.3080904%40bristol.ac.uk%3E
> I was able to reproduce the performance regression with an isolated test 
> scenario. So I recreated the components ARQ, CORE, IRI, and TDB with the SVN 
> state of May 15 9 (so svn update -r {2012-05-15} and then svn clean install)
> I then created a simple test program (attached as 
> PerformanceRegressionTest.java) which I ran 4 times in a row for each version 
> of Jena. Note that I deleted the TDB directory after the first 4 runs before 
> using the other Jena version. Attached are the files with the output. The 
> regression is obvious

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-256) Significant performance regression (TDB?) on 2.7.1 RC compared to May 15 build

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13418601#comment-13418601
 ] 

Simon Helsen commented on JENA-256:
---

Strange, so I regenerated a build from HEAD (and checked for the batch size) 
and I still get similar performances as I mentioned before. (this is using 
Oracle 6). One thing is that before I start the test, I remove the index on 
disk. I suspect that is why the first run is faster. Also, I don't understand 
your last comment "it is worth trying your example code with a singe read 
transaction around the whole loop". If I do that, I create a nested transaction 
which is not permitted. I'll try with apache-jena-2.7.3-20120718.212752-16.zip 
as well

> Significant performance regression (TDB?) on 2.7.1 RC compared to May 15 build
> --
>
> Key: JENA-256
> URL: https://issues.apache.org/jira/browse/JENA-256
> Project: Apache Jena
>  Issue Type: Bug
>  Components: TDB
>Affects Versions: Jena 2.7.1
> Environment: Windows 7, 64 bit, tested against 2.7.1 RC from June 9 
> versus a build from May 15
>Reporter: Simon Helsen
>Assignee: Andy Seaborne
>Priority: Critical
> Fix For: TDB 0.9.3
>
> Attachments: 271May15Output1.txt, 271May15Output2.txt, 
> 271May15Output3.txt, 271May15Output4.txt, 271RCOutput1.txt, 271RCOutput2.txt, 
> 271RCOutput3.txt, 271RCOutput4.txt, PerformanceRegressionTest.java
>
>
> See also 
> http://mail-archives.apache.org/mod_mbox/jena-dev/201206.mbox/%3C4FD9F19E.3080904%40bristol.ac.uk%3E
> I was able to reproduce the performance regression with an isolated test 
> scenario. So I recreated the components ARQ, CORE, IRI, and TDB with the SVN 
> state of May 15 9 (so svn update -r {2012-05-15} and then svn clean install)
> I then created a simple test program (attached as 
> PerformanceRegressionTest.java) which I ran 4 times in a row for each version 
> of Jena. Note that I deleted the TDB directory after the first 4 runs before 
> using the other Jena version. Attached are the files with the output. The 
> regression is obvious

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-283) FILTER equality transform is not handling OPTIONAL properly.

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13418437#comment-13418437
 ] 

Simon Helsen commented on JENA-283:
---

thanks for opening it. I was going to do it, but you beat me to it.

> FILTER equality transform is not handling OPTIONAL properly.
> 
>
> Key: JENA-283
> URL: https://issues.apache.org/jira/browse/JENA-283
> Project: Apache Jena
>  Issue Type: Task
>  Components: ARQ
>Affects Versions: ARQ 2.9.2
>Reporter: Andy Seaborne
>Assignee: Andy Seaborne
>Priority: Minor
>
> PREFIX  p2:   
> PREFIX  p1:   
> SELECT DISTINCT  ?R1 ?optionalValue
> WHERE
>   { ?R1 p1:pr1 
> FILTER ( ?R1 =  )
> OPTIONAL
>   { ?R1 p2:pr2 ?optionalValue }
>   }
> ==>
> (base 
>   (prefix ((p2: )
>(p1: ))
> (distinct
>   (project (?R1 ?optionalValue)
> (filter (= ?R1 )
>   (leftjoin
> (bgp (triple ?R1 p1:pr1 
> ))
> (bgp (triple ?R1 p2:pr2 ?optionalValue
> It should use substitute / extend.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-256) Significant performance regression (TDB?) on 2.7.1 RC compared to May 15 build

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13418331#comment-13418331
 ] 

Simon Helsen commented on JENA-256:
---

if the codebase has it on, I'll just regenerate a local snapshot

> Significant performance regression (TDB?) on 2.7.1 RC compared to May 15 build
> --
>
> Key: JENA-256
> URL: https://issues.apache.org/jira/browse/JENA-256
> Project: Apache Jena
>  Issue Type: Bug
>  Components: TDB
>Affects Versions: Jena 2.7.1
> Environment: Windows 7, 64 bit, tested against 2.7.1 RC from June 9 
> versus a build from May 15
>Reporter: Simon Helsen
>Assignee: Andy Seaborne
>Priority: Critical
> Fix For: TDB 0.9.3
>
> Attachments: 271May15Output1.txt, 271May15Output2.txt, 
> 271May15Output3.txt, 271May15Output4.txt, 271RCOutput1.txt, 271RCOutput2.txt, 
> 271RCOutput3.txt, 271RCOutput4.txt, PerformanceRegressionTest.java
>
>
> See also 
> http://mail-archives.apache.org/mod_mbox/jena-dev/201206.mbox/%3C4FD9F19E.3080904%40bristol.ac.uk%3E
> I was able to reproduce the performance regression with an isolated test 
> scenario. So I recreated the components ARQ, CORE, IRI, and TDB with the SVN 
> state of May 15 9 (so svn update -r {2012-05-15} and then svn clean install)
> I then created a simple test program (attached as 
> PerformanceRegressionTest.java) which I ran 4 times in a row for each version 
> of Jena. Note that I deleted the TDB directory after the first 4 runs before 
> using the other Jena version. Attached are the files with the output. The 
> regression is obvious

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-256) Significant performance regression (TDB?) on 2.7.1 RC compared to May 15 build

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13418287#comment-13418287
 ] 

Simon Helsen commented on JENA-256:
---

Hmmm, perhaps I was still running unoptimized. I used the following snapshot: 
https://repository.apache.org/content/repositories/snapshots/org/apache/jena/apache-jena/2.7.3-SNAPSHOT/apache-jena-2.7.3-20120718.050811-15.zip

Just wondering, why does the current code base have the unoptimized version by 
default? Is there a system property I can set to make sure I am using a non-0 
batch size. Also, what was the batch size you used? Does it make a difference 
what size you use?

> Significant performance regression (TDB?) on 2.7.1 RC compared to May 15 build
> --
>
> Key: JENA-256
> URL: https://issues.apache.org/jira/browse/JENA-256
> Project: Apache Jena
>  Issue Type: Bug
>  Components: TDB
>Affects Versions: Jena 2.7.1
> Environment: Windows 7, 64 bit, tested against 2.7.1 RC from June 9 
> versus a build from May 15
>Reporter: Simon Helsen
>Assignee: Andy Seaborne
>Priority: Critical
> Fix For: TDB 0.9.3
>
> Attachments: 271May15Output1.txt, 271May15Output2.txt, 
> 271May15Output3.txt, 271May15Output4.txt, 271RCOutput1.txt, 271RCOutput2.txt, 
> 271RCOutput3.txt, 271RCOutput4.txt, PerformanceRegressionTest.java
>
>
> See also 
> http://mail-archives.apache.org/mod_mbox/jena-dev/201206.mbox/%3C4FD9F19E.3080904%40bristol.ac.uk%3E
> I was able to reproduce the performance regression with an isolated test 
> scenario. So I recreated the components ARQ, CORE, IRI, and TDB with the SVN 
> state of May 15 9 (so svn update -r {2012-05-15} and then svn clean install)
> I then created a simple test program (attached as 
> PerformanceRegressionTest.java) which I ran 4 times in a row for each version 
> of Jena. Note that I deleted the TDB directory after the first 4 runs before 
> using the other Jena version. Attached are the files with the output. The 
> regression is obvious

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-256) Significant performance regression (TDB?) on 2.7.1 RC compared to May 15 build

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13417721#comment-13417721
 ] 

Simon Helsen commented on JENA-256:
---

Unfortunately, the new fixes don't seem to make a big difference (if anything 
made things a bit worse, compared to 2.7.1):

2.7.3 (July 18) - test 1:
All 100 write operations wrote 5250 triples and took 7771ms
Total opening transaction time was 235ms
Total commit transaction time was 6380ms
2.7.3 (July 18) - test 2:
All 100 write operations wrote 5250 triples and took 14248ms
Total opening transaction time was 203ms
Total commit transaction time was 12334ms
2.7.3 (July 18) - test 3:
All 100 write operations wrote 5250 triples and took 12404ms
Total opening transaction time was 183ms
Total commit transaction time was 10948ms

> Significant performance regression (TDB?) on 2.7.1 RC compared to May 15 build
> --
>
> Key: JENA-256
> URL: https://issues.apache.org/jira/browse/JENA-256
> Project: Apache Jena
>  Issue Type: Bug
>  Components: TDB
>Affects Versions: Jena 2.7.1
> Environment: Windows 7, 64 bit, tested against 2.7.1 RC from June 9 
> versus a build from May 15
>Reporter: Simon Helsen
>Assignee: Andy Seaborne
>Priority: Critical
> Fix For: TDB 0.9.3
>
> Attachments: 271May15Output1.txt, 271May15Output2.txt, 
> 271May15Output3.txt, 271May15Output4.txt, 271RCOutput1.txt, 271RCOutput2.txt, 
> 271RCOutput3.txt, 271RCOutput4.txt, PerformanceRegressionTest.java
>
>
> See also 
> http://mail-archives.apache.org/mod_mbox/jena-dev/201206.mbox/%3C4FD9F19E.3080904%40bristol.ac.uk%3E
> I was able to reproduce the performance regression with an isolated test 
> scenario. So I recreated the components ARQ, CORE, IRI, and TDB with the SVN 
> state of May 15 9 (so svn update -r {2012-05-15} and then svn clean install)
> I then created a simple test program (attached as 
> PerformanceRegressionTest.java) which I ran 4 times in a row for each version 
> of Jena. Note that I deleted the TDB directory after the first 4 runs before 
> using the other Jena version. Attached are the files with the output. The 
> regression is obvious

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (JENA-256) Significant performance regression (TDB?) on 2.7.1 RC compared to May 15 build

[Simon Helsen (JIRA)]

[ 
https://issues.apache.org/jira/browse/JENA-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13415736#comment-13415736
 ] 

Simon Helsen commented on JENA-256:
---

thanks for the fix Andy. I will try it out and see how it affects the 
performance. Provided we find this significantly improves the write 
performance, do you think it would be reasonable to request a Jena 2.7.3 
release? 

> Significant performance regression (TDB?) on 2.7.1 RC compared to May 15 build
> --
>
> Key: JENA-256
> URL: https://issues.apache.org/jira/browse/JENA-256
> Project: Apache Jena
>  Issue Type: Bug
>  Components: TDB
>Affects Versions: Jena 2.7.1
> Environment: Windows 7, 64 bit, tested against 2.7.1 RC from June 9 
> versus a build from May 15
>Reporter: Simon Helsen
>Assignee: Andy Seaborne
>Priority: Critical
> Fix For: TDB 0.9.3
>
> Attachments: 271May15Output1.txt, 271May15Output2.txt, 
> 271May15Output3.txt, 271May15Output4.txt, 271RCOutput1.txt, 271RCOutput2.txt, 
> 271RCOutput3.txt, 271RCOutput4.txt, PerformanceRegressionTest.java
>
>
> See also 
> http://mail-archives.apache.org/mod_mbox/jena-dev/201206.mbox/%3C4FD9F19E.3080904%40bristol.ac.uk%3E
> I was able to reproduce the performance regression with an isolated test 
> scenario. So I recreated the components ARQ, CORE, IRI, and TDB with the SVN 
> state of May 15 9 (so svn update -r {2012-05-15} and then svn clean install)
> I then created a simple test program (attached as 
> PerformanceRegressionTest.java) which I ran 4 times in a row for each version 
> of Jena. Note that I deleted the TDB directory after the first 4 runs before 
> using the other Jena version. Attached are the files with the output. The 
> regression is obvious

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira