Hi, On Fri, 2011-12-16 at 05:39 -0800, Milorad Tosic wrote: > Hi, > > > There is a short discussion on the former Jena Development/Support mailing > list related to memory usage of Jena OntModel using different versions of > Jena. For your information I have included a segment of the correspondence at > the end of the message. Since I had a problem that looks much like memory > usage limit overrun, I got interested in the problem. > > > I repeated the same experiment but with > different conclusions? I compared Jena 2.6.0 with Jena 2.6.4. I couldn't > notice the difference between Jena versions as reported in your > previous experiment (though I used the same code). Tried with several > variations of the OntModel creations styles but with minor differences > in memory usage only (within 10%). I run the experiment from within > Eclipse with JVM set to jre6 using compiler compliance level 1.5 (as > well as 1.6). The total memory usage was around 27975k. > > Than, I changed the JVM to jdk 1.5. There was similar relative > difference between the Jena 2.6.0 and Jena 2.6.4 (within 10%). However, > the absolute amount of used memory dropped to about 3400k. > > Than I again changed the JVM to jdk 1.6. As in all previous testing > configurations, there was similar relative difference between the Jena > 2.6.0 and Jena 2.6.4 (within 10%). However, the absolute amount of used > memory was about 6501k. > > Consequently, I have two problems now: > > 1) I couldn't repeat you experiment with the same results. What I am doing > wrong?
Assuming you are running the code that Rick originally posted then the problem is that that uses a naive way to measure memory usage. If I run with java 1.6 (sun-java on Linux) and the current Jena snapshot and run in eclipse I get: time: 19, totMem: 122688k, usedMem: 4859k If I run identical code from the command line I get: time: 12, totMem: 122688k, usedMem: 28548k The difference is down to whether a garbage collect has kicked in. If I force a System.gc() in that code then I get: time: 86, totMem: 122688k, usedMem: 4006k where the extra time comes from the garbage collect. > 2) What should I do to put the memory usage under my control? Namely, 5 > times larger memory usage cause my application to broke on machines > with limited memory usage. I don't think there's any evidence that the problem in the thread that you are referring to is still an issue. You would need to explain more about what you are actually do in your application before we could help with that. However, if you are currently using inference and are having memory problems then avoiding inference, or using a weaker inference setting, would be a good place to start. Dave > > Thanks, > Milorad Tosic > > > > --- In [email protected], Dave Reynolds <dave.e.reynolds@...> wrote: > > > > > > Rick, > > > > > > Thanks for letting us know, thanks for the clear report. > > > > > > Dave > > > > > > rspates2 wrote: > > > > Dave, > > > > > > > > > I grabbed OWLProfile.java and reran my tests. It looks fixed > to me in terms of speed and memory use. We should be able to use 2.6.2 > now. > > > > > > > > Thanks for your swift resolution to the problem. > > > > > > > > Rick > > > > > > > > --- In [email protected], Dave Reynolds <dave.e.reynolds@> wrote: > > > >> Hi Rick, > > > >> > > > >> A follow up on my earlier off-list response. > > > >> > > > >> Thanks again for giving us a test case for this. It turns out not to > > > >> be > > > >> a reasoner issue as such. There was a change in the internals of the > > > >> Ontology API which worked inefficiently when a reasoner is present. > > > >> Ian > > > >> has checked in some patches for this which brings the memory use down > > > >> a > > > >> lot and gives a 5x speed improvement. > > > >> > > > >> It is not quite back to 2.6.0 timings but is a significant improvement. > > > >> > > > >> We'll investigate a little more but this may be as close as we can get > > > >> for now. Let us know if this is enough to make it usable for you. > > > >> > > > >> Cheers, > > > >> Dave > > > >> > > > >> rspates2 wrote: > > > >>> Dave, > > > >>> > > > >>> The difference seems to be between 2.6.0 and 2.6.2, > > > >>> > > > >>> OntModel.listClasses(): > > > >>> 2.6.2 time: 2187, totMem: 19948k, usedMem: 13345k > > > >>> 2.6.0 time: 110, totMem: 5056k, usedMem: 3681k > > > >>> 2.5.7 time: 78, totMem: 5056k, usedMem: 3726k > > > >>> > > > >>> listHierarchyRootClasses(): > > > >>> 2.6.2 time: 2312, totMem: 20152k, usedMem: 13859k > > > >>> 2.6.0 time: 172, totMem: 5056k, usedMem: 4312k > > > >>> 2.5.7 time: 188, totMem: 5056k, usedMem: 4078k > > > >>> > > > >>> Let me know if you need anything else. > > > >>> > > > >>> Thanks, > > > >>> Rick > > > >>> > > > >>> --- In [email protected], Dave Reynolds <dave.e.reynolds@> > > > >>> wrote: > > > >>>> rspates2 wrote: > > > >>>>> Hi, > > > >>>>> > > > > >>>>> We switched our application from Jena 2.5.7 > to 2.6.2. For an operation on OntModel.listHierarchyRootClasses() with > an OntModelSpec of OWL_MEM_RDFS_INF, we saw a very significant increase > in the use of memory and processing time, including out-of-memory > condition for our full ontology. Using a subset of our ontology and the > code below to demonstrate, we measured processing times and memory use: > > > >>>>> > > > >>>>> 2.6.2 time: 2063, totMem: 20008k, usedMem: 14539k > > > >>>>> 2.5.7 time: 172, totMem: 5056k, usedMem: 3974k > > > >>>> Certainly a huge jump :( > > > >>>> > > > > >>>>> The specifications OWL_MEM_RDFS_INF and > OWL_DL_MEM_RDFS_INF produced comparable results, while > OWL_DL_MEM_TRANS_INF at 2.6.2 was comparable to our 2.5.7 run with > OWL_MEM_RDFS_INF. > > > >>>>> > > > > >>>>> I can post the subset ontology if you need it (170 > k), but thought there might be some known explanation you could provide > without running the demo code. > > > >>>>> > > > >>>>> Any idea what is causing this? > > > >>>> No immediate ideas. The RFDS rule file itself hasn't changed in that > > > >>>> time frame. It is tricky to trace back code changes over that period > > > >>>> because we did a code reorg which hides a lot of the history. > > > >>>> However, > > > >>>> from memory (and the release notes) I can't think of anything that > > > >>>> should have had an effect like that. There were changes around > > > >>>> handling > > > >>>> validation of data types but that isn't relevant here. We > > > >>>> generalized > > > >>>> handling of non-legal RDF triples but again that shouldn't be > > > >>>> relevant. > > > >>>> > > > >>>> Could you try two experiments? > > > >>>> > > > >>>> First, would it be possible to time a simpler operation, such as > > > >>>> listing > > > >>>> all classes, to eliminate listHierarchyRootClasses as a factor? > > > >>>> > > > >>>> Secondly, is it possible to check with Jena 2.6.0 - that would > > > >>>> separate > > > >>>> the Java 5 changes and the datatype validation from more recent > > > >>>> tweaks? > > > >>>> > > > >>>> If that's a problem then send me your ontology subset and I'll take > > > >>>> a > > > >>>> look when I can. > > > >>>> > > > >>>> Dave > > > >>>> > > > >>>>> Thanks for your help, > > > >>>>> Rick > > > >>>>> > > > >>>>> import java.io.InputStream; > > > >>>>> import org.junit.Test; > > > >>>>> > > > >>>>> import com.hp.hpl.jena.ontology.OntModel; > > > >>>>> import com.hp.hpl.jena.ontology.OntModelSpec; > > > >>>>> import com.hp.hpl.jena.rdf.model.Model; > > > >>>>> import com.hp.hpl.jena.rdf.model.ModelFactory; > > > >>>>> import com.hp.hpl.jena.util.iterator.ExtendedIterator; > > > >>>>> > > > >>>>> public class TestReasonerProblem { > > > >>>>> > > > >>>>> @Test > > > >>>>> public void test() throws Exception { > > > >>>>> > > > >>>>> Model model = ModelFactory.createDefaultModel(); > > > >>>>> String file="clinical-factDCMapVent.owl"; > > > >>>>> InputStream fis = > > > >>>>> this.getClass().getResourceAsStream(file); > > > >>>>> model.read(fis, null, "RDF/XML"); > > > >>>>> OntModelSpec spec = OntModelSpec.OWL_MEM_RDFS_INF; > > > >>>>> // OntModelSpec spec = OntModelSpec.OWL_DL_MEM_RDFS_INF; > > > >>>>> // OntModelSpec spec = OntModelSpec.OWL_DL_MEM_TRANS_INF; > > > >>>>> OntModel ontModel = ModelFactory.createOntologyModel(spec, > > > >>>>> model); > > > >>>>> > > > >>>>> long ctms = System.currentTimeMillis(); > > > >>>>> for (ExtendedIterator j = > > > >>>>> ontModel.listHierarchyRootClasses(); j.hasNext(); ){ > > > >>>>> j.next(); > > > >>>>> } > > > >>>>> System.out.printf("time: %d, totMem: %dk, usedMem: %dk\n", > > > >>>>> (System.currentTimeMillis()-ctms), > > > >>>>> (Runtime.getRuntime().totalMemory()/1024), > > > >>>>> > > > >>>>> ((Runtime.getRuntime().totalMemory()-Runtime.getRuntime().freeMemory())/1024)); > > > >>>>> } > > > >>>>> > > > >>>>> } > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>> ------------------------------------ > > > >>>>> > > > >>>>> Yahoo! Groups Links > > > >>>>> > > > >>>>> > > > >>>>> > > > >>> > > > >>> > > > >>> > > > >>> ------------------------------------ > > > >>> > > > >>> Yahoo! Groups Links > > > >>> > > > >>> > > > >>> > > > > > > > > > > > > > > > > > > > > ------------------------------------ > > > > > > > > Yahoo! Groups Links > > > > > > > > > > > > > > > > > > >
