StackOverflow said - rightly or wrongly - that one is always just running a 32-big JVM (Under Windows).
Lorenz suggested omitting convert_geo param which seemed to push it through. On Thu, 15 Jul 2021 at 19:19, Andy Seaborne <[email protected]> wrote: > -Xmx1000m is 1Gbytes - that's small. > > Are running a 32 bit java? There is a comment on StackOverflow comment > saying you are. The maximum possible heap size is about 1.5G. > > In 32 bit mode, TDB can't cache in the same way. It will be slow. > > You also you dropped the -i (RDFS inference) which also reduces the > memory footprint. > > Andy > > System java for me: > openjdk version "17-ea" 2021-09-14 > OpenJDK Runtime Environment (build 17-ea+19-Ubuntu-1ubuntu1) > OpenJDK 64-Bit Server VM (build 17-ea+19-Ubuntu-1ubuntu1, mixed mode, > sharing) > > Even Java8: > openjdk version "1.8.0_292" > OpenJDK Runtime Environment (build 1.8.0_292-8u292-b10-0ubuntu1-b10) > OpenJDK 64-Bit Server VM (build 25.292-b10, mixed mode) > > > On 15/07/2021 13:47, Matt Whitby wrote: > > I tried it without the --convert_geo param, and with -Xmx1000m for the > heap > > size and that did seem to fix it. > > > > Thank you. > > > > java -jar -Xmx1000m jena-fuseki-geosparql-3.17.0.jar --tdb spatialdb2 > > > > > > > > > > On Thu, 15 Jul 2021 at 13:12, Lorenz Buehmann < > > [email protected]> wrote: > > > >> 1.) do you need convert_geo param? > >> > >> 2.) Xss does increase the stack size, if you run out of memory you > >> should increase Xmx to e.g. 4G or something depending on your machine. > >> If the geospatial index has been generated I'm wondering what is done > >> in-memory, so if you don't need the convert_geo option, you should omit > >> it and try again > >> > >> On 15.07.21 13:28, Matt Whitby wrote: > >>> Hi Andy, Lorenz.. > >>> > >>> I tried the TDBLoader and it generated the indexes, etc. > >>> > >>> image.png > >>> > >>> > >>> boot up geosparql with the tdb file and I run out of memory too. > >>> > >>> java -jar jena-fuseki-geosparql-3.17.0.jar --convert_geo --tdb > >>> spatialdb2 -i > >>> > >>> /Exception in thread "main" java.lang.OutOfMemoryError: Java heap > space/ > >>> > >>> Now, I'm sure the solution is to add a param akin to *-Xss8m* to book > >>> the JVM with more memory. > >>> > >>> java -jar *-Xss8m* jena-fuseki-geosparql-3.17.0.jar --convert_geo > >>> --tdb spatialdb2 > >>> > >>> Does this sound right to you? > >>> > >>> > >>> > >>> On Tue, 13 Jul 2021 at 22:09, Andy Seaborne <[email protected] > >>> <mailto:[email protected]>> wrote: > >>> > >>> > >>> > >>> On 13/07/2021 11:31, Matt Whitby wrote: > >>> > Morning all. > >>> > > >>> > > >>> > It's just on my laptop (though with 64gb of memory so more than > >>> enough I > >>> > would assume). > >>> > >>> Unless you have changed something, it is using 16G. > >>> > >>> However, it is not clear that it is a memory space issue. > >>> > >>> > > >>> > The file is about 850mb, so not that big in the scheme of > things. > >>> > > >>> > I don't see any log files. > >>> > >>> They have been printed to stdout as shown. They can go elsewhere > >>> (it's > >>> log4j2). > >>> > >>> > > >>> > The full stack trace is... > >>> > >>> There was an Java Error (the code doesn't print it - an > >>> oversight). Out > >>> of memory is an error. > >>> > >>> How long after the "DatasetOperations :: Reading RDF - Started - > >>> File:" > >>> output does it fail? > >>> > >>> It is worth checking the file parses correctly. e.g. some encoding > >>> errors become Java "errors" in 3.17.0. > >>> > >>> > 11:26:01 INFO DatasetOperations :: In-Memory Dataset > >>> > >>> That means there are multiple copies in-memory during loading. > >>> This does not explain using 16G. > >>> > >>> But the database can be loaded using the TDB bulkloader separately > >>> from > >>> the server starting and then pass in the persistent database so > >>> the file > >>> does not have to be read each start-up. > >>> > >>> Andy > >>> > > >>> > C:\Data\apache-jena-fuseki-3.17.0>java -jar > >>> > jena-fuseki-geosparql-3.17.0.jar --convert_geo -rf > >>> "nhle_spatial3.ttl" -i > >>> > > >>> > 11:26:01 INFO Main :: Arguments Received: > >>> [--convert_geo, -rf, > >>> > nhle_spatial3.ttl, -i] > >>> > 11:26:01 INFO DatasetOperations :: Server Configuration: > >> port=3030, > >>> > datsetName=ds, loopbackOnly=true, updateAllowed=false, > >>> inference=true, > >>> > applyDefaultGeometry=false, validateGeometryLiteral=false, > >>> > convertGeoPredicates=true, removeGeoPredicates=false, > >>> queryRewrite=true, > >>> > tdbFile=null, > >>> fileGraphFormats=[FileGraphFormat{rdfFile=nhle_spatial3.ttl, > >>> > graphName=, rdfFormat=Turtle/pretty}], fileGraphDelimiters=[], > >>> > indexEnabled=true, indexSizes=[-1, -1, -1], indexExpiries=[5000, > >>> 5000, > >>> > 5000], spatialIndexFile=null, tdb2=false, help=false > >>> > 11:26:01 INFO DatasetOperations :: In-Memory Dataset > >>> > 11:26:02 INFO DatasetOperations :: Reading RDF - Started - > File: > >>> > nhle_spatial3.ttl, Graph Name: , RDF Format: Turtle/pretty > >>> > 11:26:02 WARN system :: The ôSIS_DATAö environment > >>> variable is > >>> > not set. > >>> > Exception in thread "main" > >>> org.apache.jena.sparql.JenaTransactionException: > >>> > Write transaction - no commit or abort before end() > >>> > at > >>> > > >>> > >> > > org.apache.jena.sparql.core.TransactionalLock.error(TransactionalLock.java:179) > >>> > at > >>> > > >>> > >> > org.apache.jena.sparql.core.TransactionalLock.end(TransactionalLock.java:162) > >>> > at > >>> > > >>> > >> > org.apache.jena.sparql.core.DatasetGraphMap.end(DatasetGraphMap.java:80) > >>> > at > >>> org.apache.jena.sparql.core.DatasetImpl.end(DatasetImpl.java:164) > >>> > at > >>> > > >>> > >> > > org.apache.jena.fuseki.geosparql.DatasetOperations.loadData(DatasetOperations.java:170) > >>> > at > >>> > > >>> > >> > > org.apache.jena.fuseki.geosparql.DatasetOperations.setup(DatasetOperations.java:68) > >>> > at > >> org.apache.jena.fuseki.geosparql.Main.main(Main.java:64) > >>> > > >>> > > >>> > > >>> > > >>> > On Mon, 12 Jul 2021 at 20:32, Andy Seaborne <[email protected] > >>> <mailto:[email protected]>> wrote: > >>> > > >>> >> There would have been more logging and more stack trace > >>> >> > >>> >> but it looks like you are loading all the data into memory > >>> >> Did it log "In-Memory Dataset"? > >>> >> > >>> >> How big is nhle_spatial5.ttl? How big is the machine it is > >>> running on? > >>> >> > >>> >> Andy > >>> >> > >>> >> On 12/07/2021 14:44, Matt Whitby wrote: > >>> >>> Good afternoon all. > >>> >>> > >>> >>> I've been trying to import a TTL file into GeoSparql. It's > >>> worked fine > >>> >> for > >>> >>> smaller datasets, but now they're getting bigger (the > >>> production one > >>> >> would > >>> >>> be about 900mb) I'm getting an error. > >>> >>> > >>> >>> java -jar jena-fuseki-geosparql-3.17.0.jar --convert_geo -rf > >>> >>> "nhle_spatial5.ttl" -i > >>> >>> > >>> >>> Error: > >>> >>> > >>> >>> Reading RDF - Started - File: nhle_spatial5.ttl, Graph Name: , > >> RDF > >>> >> Format: > >>> >>> Turtle/pretty > >>> >>> Exception in thread "main" > >>> >> org.apache.jena.sparql.JenaTransactionException: > >>> >>> Write transaction - no commit or abort before end() > >>> >>> at > >>> >>> > >>> >> > >>> > >> > > org.apache.jena.sparql.core.TransactionalLock.error(TransactionalLock.java:179) > >>> >>> > >>> >>> Might I be correct in thinking it's a size issue? > >>> >>> > >>> >>> > >>> >>> Kind Regards, > >>> >>> M > >>> >>> > >>> >> > >>> > > >>> > > >>> > >>> > >>> > >>> -- > >>> Matt > >>> Southend. Essex, England > >>> > >>> Guff follows.... > >>> > >>> Me: http://www.about.me/matt.whitby <http://www.about.me/matt.whitby> > >>> > >>> > >>> Photography: http://www.whitbyphoto.com <http://www.whitbyphoto.com/> > >>> > >>> > >>> Travels: http://www.whitbyadventures.com > >>> <http://www.whitbyadventures.com/> > >>> > >>> > >>> Music: http://www.last.fm/user/MattWhitby > >>> <http://www.last.fm/user/MattWhitby/%3C/a%3E> > >>> > >>> > >>> Reading: https://www.goodreads.com/user_challenges/19398505 > >>> <https://www.goodreads.com/user_challenges/19398505> > >>> > >>> > >>> Development: https://www.hackerrank.com/matt_whitby > >>> <https://www.hackerrank.com/matt_whitby> > >>> > >> > > > > > -- Matt Southend. Essex, England Guff follows.... Me: http://www.about.me/matt.whitby Photography: http://www.whitbyphoto.com Travels: http://www.whitbyadventures.com Music: http://www.last.fm/user/MattWhitby <http://www.last.fm/user/MattWhitby/%3C/a%3E> Reading: https://www.goodreads.com/user_challenges/19398505 Development: https://www.hackerrank.com/matt_whitby
