Simon
From:
Andy Seaborne<[email protected]>
To:
[email protected]
Date:
09/15/2011 09:10 AM
Subject:
Re: [jira] [Commented] (JENA-91) extremely large buffer is being created
in ObjectFileStorage
(nothing attached)
I can recreate on windows/64, not linux (only tried 64 bit) I have not
tried (windows|linux)/32. Some a few tests of the overall datasets
change by architecture but most do not - the current failures are neutral.
I do have to use real hardware, not a VM, which slows investigation time
down. (This is unrelated - I have never got a VirtualBox particular
stable with a 64bit OS; I don't usually even get to the tests.)
Some of the errors are spurious - failing because windows handling of
mmaped files and delete. These can be coded around and I'll check in
some mods for that.
TestBlockAccessMapped
fileaccess_03, fileaccess_04 (these were reusing a mapped file)
and maybe some, but not all, of the others.
Simon - the patch file TestTransSystem4.patch is not marked as licensed
for Apache works.
Andy
On 14/09/11 22:25, Simon Helsen wrote:
perhaps it is considered costly and not everyone needs it. Anyhow, here
it is:
From: Andy Seaborne<[email protected]>
To: [email protected]
Date: 09/14/2011 05:20 PM
Subject: Re: [jira] [Commented] (JENA-91) extremely large
buffer is
being created in ObjectFileStorage
------------------------------------------------------------------------
Simon,
Thanks, I can produce them from those file (not sure why the normal
build does not do it, it's
mvn surefire-report:report-only
)
Andy
On 14/09/11 22:11, Simon Helsen wrote:
> Andy,
>
> keep in mind that my patched version of TestTransSystem also fails in
> direct mode so please try that as well in case the regular tests only
> fail because of known limitations. We (IBM) only use direct mode and
my
> internal tests fail equally in direct mode.
> Here is the report (not sure about html, it only contains .txt and
.xml)
>
>
>
> Simon
>
>
>
> From: Andy Seaborne<[email protected]>
> To: [email protected]
> Date: 09/14/2011 05:02 PM
> Subject: Re: [jira] [Commented] (JENA-91) extremely large buffer is
> being created in ObjectFileStorage
>
>
>
------------------------------------------------------------------------
>
>
>
> One note: some of the tests are known to fail on Windows 64 bit
because
> windows does not release memory mapped files in the same way Linux
does
> (Linux does it mid-run). That might cover some of the test failures
but
> not all of them.
>
> I'll find a windows/64 machine.
>
> Simon - could you send me the surefire HTML report please?
>
> Andy
>
>
>
> On 14/09/11 19:04, Simon Helsen wrote:
> > actually, I just executed exactly what you (Paolo) did, but on
> Windows. My
> > results are not quite like what Dave saw, but they fail
nonetheless. I'll
> > add this to Jira
> >
> > Here is the output:
> >
> > D:\jfsDev\TxTDB>svn up
> > At revision 1170749.
> >
> > D:\jfsDev\TxTDB>mvn clean test
> > [INFO] Scanning for projects...
> > [INFO]
> > [INFO]
> >
------------------------------------------------------------------------
> > [INFO] Building TxTDB 0.9.0-SNAPSHOT
> > [INFO]
> >
------------------------------------------------------------------------
> > [INFO]
> > [INFO] --- maven-clean-plugin:2.4.1:clean (default-clean) @ tx-tdb
---
> > [INFO] Deleting D:\jfsDev\TxTDB\target
> > [INFO]
> > [INFO] --- maven-antrun-plugin:1.3:run (ant-timestamp) @ tx-tdb ---
> > [INFO] Executing tasks
> > [mkdir] Created dir: D:\jfsDev\TxTDB\target
> > [echo] version=0.9.0-SNAPSHOT
> > [echo] build.time=2011-09-14 14:00 EDT
> > [INFO] Executed tasks
> > [INFO]
> > [INFO] --- maven-resources-plugin:2.5:resources (default-resources)
@
> > tx-tdb ---
> >
> > [debug] execute contextualize
> > [INFO] Using 'UTF-8' encoding to copy filtered resources.
> > [INFO] Copying 1 resource to com/hp/hpl/jena/tdb/
> > [INFO]
> > [INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @
tx-tdb
> > ---
> > [INFO] Compiling 268 source files to D:\jfsDev\TxTDB\target\classes
> > [INFO]
> > [INFO] --- maven-resources-plugin:2.5:testResources
> > (default-testResources) @ tx
> > -tdb ---
> > [debug] execute contextualize
> > [INFO] Using 'UTF-8' encoding to copy filtered resources.
> > [INFO] skip non existing resourceDirectory
> > D:\jfsDev\TxTDB\src\test\resources
> > [INFO]
> > [INFO] --- maven-compiler-plugin:2.3.2:testCompile
> (default-testCompile) @
> > tx-td
> > b ---
> > [INFO] Compiling 115 source files to
D:\jfsDev\TxTDB\target\test-classes
> > [INFO]
> > [INFO] --- maven-surefire-plugin:2.9:test (default-test) @ tx-tdb
---
> > [INFO] Surefire report directory:
D:\jfsDev\TxTDB\target\surefire-reports
> >
> > -------------------------------------------------------
> > T E S T S
> > -------------------------------------------------------
> > Running com.hp.hpl.jena.tdb.assembler.TS_TDBAssembler
> > Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
2.087 sec
> > Running com.hp.hpl.jena.tdb.base.block.TS_Block
> > Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 0.803 sec
> > Running com.hp.hpl.jena.tdb.base.file.TS_File
> > Tests run: 39, Failures: 2, Errors: 0, Skipped: 0, Time elapsed:
> 0.054 sec
> > <<< F
> > AILURE!
> > Running com.hp.hpl.jena.tdb.base.objectfile.TS_ObjectFile
> > Tests run: 48, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 0.091 sec
> > Running com.hp.hpl.jena.tdb.base.record.TS_Record
> > Tests run: 54, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 0.047 sec
> > Running com.hp.hpl.jena.tdb.base.recordfile.TS_RecordFile
> > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
0.002 sec
> > Running com.hp.hpl.jena.tdb.graph.TS_Graph
> > Tests run: 40, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 0.674 sec
> > Running com.hp.hpl.jena.tdb.index.TS_Index
> > Tests run: 97, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 0.242 sec
> > Running com.hp.hpl.jena.tdb.lib.TS_LibTDB
> > Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
0.009 sec
> > Running com.hp.hpl.jena.tdb.migrate.TS_Migrate
> > Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
0.01 sec
> > Running com.hp.hpl.jena.tdb.nodetable.TS_NodeTable
> > Tests run: 37, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
0.05 sec
> > Running com.hp.hpl.jena.tdb.solver.TS_SolverTDB
> > Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
0.077 sec
> > Running com.hp.hpl.jena.tdb.store.TS_Store
> > Tests run: 181, Failures: 0, Errors: 2, Skipped: 0, Time elapsed:
2.094
> > sec<<<
> > FAILURE!
> > Running com.hp.hpl.jena.tdb.sys.TS_Sys
> > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
0.013 sec
> > Running com.hp.hpl.jena.tdb.transaction.TS_Transaction
> > Tests run: 77, Failures: 13, Errors: 0, Skipped: 0, Time elapsed:
3.569
> > sec<<<
> > FAILURE!
> > Running com.hp.hpl.jena.tdb.TS_Factory
> > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0
sec
> > Running com.hp.hpl.jena.tdb.TS_Jena
> > Tests run: 73, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 0.096 sec
> >
> > Results :
> >
> > Failed tests:
> > fileaccess_03(com.hp.hpl.jena.tdb.base.file.TestBlockAccessMappe
> > d)
> > fileaccess_04(com.hp.hpl.jena.tdb.base.file.TestBlockAccessMapped)
> >
nodetrans_03(com.hp.hpl.jena.tdb.transaction.TestNodeTableTransDisk):
> > expected
> > :<[DoesNotExist]> but was:<[0000000000000000]>
> > trans_04(com.hp.hpl.jena.tdb.transaction.TestTransSequentialDisk)
> > trans_06(com.hp.hpl.jena.tdb.transaction.TestTransSequentialDisk)
> >
> >
>
trans_readBlock_01(com.hp.hpl.jena.tdb.transaction.TestTransSequentialDisk)
> >
> >
>
trans_readBlock_02(com.hp.hpl.jena.tdb.transaction.TestTransSequentialDisk)
> >
> >
>
trans_readBlock_03(com.hp.hpl.jena.tdb.transaction.TestTransSequentialDisk)
> >
> >
>
trans_readBlock_04(com.hp.hpl.jena.tdb.transaction.TestTransSequentialDisk)
> >
> >
>
trans_readBlock_05(com.hp.hpl.jena.tdb.transaction.TestTransSequentialDisk)
> >
> >
>
trans_readBlock_06(com.hp.hpl.jena.tdb.transaction.TestTransSequentialDisk)
> >
> >
>
trans_readBlock_07(com.hp.hpl.jena.tdb.transaction.TestTransSequentialDisk)
> >
> >
>
trans_readBlock_08(com.hp.hpl.jena.tdb.transaction.TestTransSequentialDisk)
> >
> >
>
trans_readBlock_09(com.hp.hpl.jena.tdb.transaction.TestTransSequentialDisk)
> >
> >
>
trans_readBlock_11(com.hp.hpl.jena.tdb.transaction.TestTransSequentialDisk)
> >
> > Tests in error:
> > dataset2(com.hp.hpl.jena.tdb.store.TestDatasetTDBPersist)
> > dataset3(com.hp.hpl.jena.tdb.store.TestDatasetTDBPersist)
> >
> > Tests run: 697, Failures: 15, Errors: 2, Skipped: 0
> >
> > [INFO]
> >
------------------------------------------------------------------------
> > [INFO] BUILD FAILURE
> > [INFO]
> >
------------------------------------------------------------------------
> > [INFO] Total time: 25.664s
> > [INFO] Finished at: Wed Sep 14 14:00:52 EDT 2011
> > [INFO] Final Memory: 30M/83M
> > [INFO]
> >
------------------------------------------------------------------------
> > [ERROR] Failed to execute goal
> > org.apache.maven.plugins:maven-surefire-plugin:2.
> > 9:test (default-test) on project tx-tdb: There are test failures.
> > [ERROR]
> > [ERROR] Please refer to D:\jfsDev\TxTDB\target\surefire-reports for
the
> > individu
> > al test results.
> > [ERROR] -> [Help 1]
> > [ERROR]
> > [ERROR] To see the full stack trace of the errors, re-run Maven
with the
> > -e swit
> > ch.
> > [ERROR] Re-run Maven using the -X switch to enable full debug
logging.
> > [ERROR]
> > [ERROR] For more information about the errors and possible
solutions,
> > please rea
> > d the following articles:
> > [ERROR] [Help 1]
> > http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureExc
> > eption
> > D:\jfsDev\TxTDB>
> >
> >
> >
> > From:
> > Paolo Castagna<[email protected]>
> > To:
> > [email protected]
> > Date:
> > 09/14/2011 01:45 PM
> > Subject:
> > Re: [jira] [Commented] (JENA-91) extremely large buffer is being
created
> > in ObjectFileStorage
> >
> >
> >
> > Hi Dave,
> > thanks (it helps).
> >
> > If there are tests failing on Windows 7, it's bad (and good at
> > the same time that the test suite is signaling the problem).
> >
> > Just to double check there are no changes in TxTDB trunk that
> > I do not have on my local copy, I've just did:
> >
> > cd /tmp/txtdb/
> > svn up
> > At revision 1170728.
> >
> > mvn clean test
> >
> > [INFO] Scanning for projects...
> > [INFO]
> >
------------------------------------------------------------------------
> > [INFO] Building TxTDB
> > [INFO] task-segment: [clean, test]
> > [INFO]
> >
------------------------------------------------------------------------
> > [...]
> >
> > Tests run: 697, Failures: 0, Errors: 0, Skipped: 0
> >
> > [INFO]
> >
------------------------------------------------------------------------
> > [INFO] BUILD SUCCESSFUL
> > [INFO]
> >
------------------------------------------------------------------------
> > [INFO] Total time: 1 minute 13 seconds
> > [INFO] Finished at: Wed Sep 14 18:36:44 BST 2011
> > [INFO] Final Memory: 42M/296M
> > [INFO]
> >
------------------------------------------------------------------------
> >
> > There is obviously something bad going when we run TxTDB on
> > Windows 7. I am afraid I cannot be more helpful on this. :-(
> >
> > @Andy, do you have a Windows 7 machine to investigate the
> > problem that Simon and Dave are experiencing?
> >
> > Paolo
> >
> > Dave Reynolds wrote:
> >> On Wed, 2011-09-14 at 16:38 +0100, Paolo Castagna wrote:
> >>> Hi Simon,
> >>> indeed, very strange.
> >>>
> >>> I do not have a Windows 7 machine to try, but if you have a Linux
> > machine
> >>> you could try there, it would be really interesting to know the
result.
> >>>
> >>> Could it be something related to your Maven cache and somehow you
still
> > have
> >>> an old ARQ SNAPSHOT? You could try to manually delete the ARQ
artifacts
> > from
> >>> your Maven cache (usually in
~/.m2/repository/com/hp/hpl/jena/arq)
> >>>
> >>> This is how you or others could run the test on different
environments:
> >>>
> >>> cd /tmp/
> >>> svn co
> >
http://svn.apache.org/repos/asf/incubator/jena/Experimental/TxTDB/trunk/
> >>> txtdb
> >>> cd /tmp/txtdb/
> >>> wget
> >>>
> >
>
https://issues.apache.org/jira/secure/attachment/12494332/TestTransSystem4.patch
> >
> >>> patch -p0< TestTransSystem4.patch
> >>> mvn package
> >>> mvn exec:java -Dexec.classpathScope="test"
> >>>
-Dexec.mainClass="com.hp.hpl.jena.tdb.transaction.TestTransSystem"
> >>>
> >>> (you need to adapt things to Windows)
> >>
> >> FWIW I get errors trying that sequence on Windows.
> >>
> >> Environment:
> >> Windows 7 64 bit
> >> Java 1.6.0_26-b03
> >>
> >> START (disk, 100 iterations)
> >> 000: ..java.lang.UnsupportedOperationException: Quad: predicate
cannot
> >> be null
> >> at com.hp.hpl.jena.sparql.core.Quad.<init>(Quad.java:49)
> >> at com.hp.hpl.jena.tdb.lib.TupleLib.quad(TupleLib.java:127)
> >> at com.hp.hpl.jena.tdb.lib.TupleLib.quad(TupleLib.java:118)
> >> at com.hp.hpl.jena.tdb.lib.TupleLib.access$100(TupleLib.java:32)
> >> at com.hp.hpl.jena.tdb.lib.TupleLib$4.convert(TupleLib.java:76)
> >> at com.hp.hpl.jena.tdb.lib.TupleLib$4.convert(TupleLib.java:72)
> >> at org.openjena.atlas.iterator.Iter$4.next(Iter.java:268)
> >> at com.hp.hpl.jena.tdb.store.GraphTDBBase
> >> $ProjectQuadsToTriples.next(GraphTDBBase.java:183)
> >> at com.hp.hpl.jena.tdb.store.GraphTDBBase
> >> $ProjectQuadsToTriples.next(GraphTDBBase.java:171)
> >> at
> >>
> >
>
com.hp.hpl.jena.util.iterator.WrappedIterator.next(WrappedIterator.java:68)
> >> at
> >>
com.hp.hpl.jena.util.iterator.Map1Iterator.next(Map1Iterator.java:35)
> >> at
> >>
> >
>
com.hp.hpl.jena.util.iterator.WrappedIterator.next(WrappedIterator.java:68)
> >> at
> >>
> >
>
com.hp.hpl.jena.rdf.model.impl.StmtIteratorImpl.next(StmtIteratorImpl.java:33)
> >> at
> >>
> >
>
com.hp.hpl.jena.rdf.model.impl.StmtIteratorImpl.next(StmtIteratorImpl.java:21)
> >> at
> >>
> >
>
com.hp.hpl.jena.tdb.transaction.TestTransSystem.readModel(TestTransSystem.java:446)
> >> at com.hp.hpl.jena.tdb.transaction.TestTransSystem.access
> >> $100(TestTransSystem.java:59)
> >> at com.hp.hpl.jena.tdb.transaction.TestTransSystem
> >> $Reader.call(TestTransSystem.java:163)
> >> at java.util.concurrent.FutureTask
> >> $Sync.innerRun(FutureTask.java:303)
> >> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> >> at java.util.concurrent.ThreadPoolExecutor
> >> $Worker.runTask(ThreadPoolExecutor.java:886)
> >> at java.util.concurrent.ThreadPoolExecutor
> >> $Worker.run(ThreadPoolExecutor.java:908)
> >> at java.lang.Thread.run(Thread.java:619)
> >>
> >>
> >> However, I get some test errors during the build and had to force
Maven
> >> to ignore those in order to get a build, so this is not a clean
run.
> >>
> >> Surefire logs for the test-during-build errors are attached.
> >>
> >> This is not my normal dev environment so there is little chance of
stale
> >> maven cache.
> >>
> >> Dave
> >>
> >>
> >>
> >
> >
> >
> >
>
>
