Hi Laura,
Thank you for sharing your experience! I think your example really shows
the power - and limitations - of HDT technology for querying very large
RDF data sets. While I don't currently have any use case for a local,
queryable Wikidata dump, I can easily see that it could be very useful
for doing e.g. resource-intensive, analytic queries. Having access to a
recent hdt+index dump of Wikidata would make it very easy to start doing
that. So I second your plea.
-Osma
Laura Morales kirjoitti 03.11.2017 klo 09:48:
Hello list,
a very kind person from this list has generated the .hdt.index file for me, using the
1-year old wikidata HDT file available at the rdfhdt website. So I was finally able to
setup a working local endpoint using HDT+Fuseki. Set up was easy, launch time (for
Fuseki) also was quick (a few seconds), the only change I made was to replace -Xmx1024m
to -Xmx4g in the Fuseki startup script (btw I'm not very proficient in Java, so I hope
this is the correct way). I've ran some queries too. Simple select or traversal queries
seems fast to me (I haven't measured them but the response is almost immediate), other
queries such as "select distinct ?class where { [] a ?class }" takes several
seconds or a few minutes to complete, which kinda tells me the HDT indexes don't work
well on all queries. But otherwise for simple queries it works perfectly! At least I'm
able to query the dataset!
In conclusion, I think this more or less gives some positive feedback for using HDT on a
"commodity computer", which means it can be very useful for people like me who
want to use the dataset locally but who can't setup a full-blown server. If others want
to try as well, they can offer more (hopefully positive) feedback.
For all of this, I heartwarmingly plea any wikidata dev to please consider
scheduling a HDT dump (.hdt + .hdt.index) along with the other regular dumps
that it creates weekly.
Thank you!!
_______________________________________________
Wikidata mailing list
Wikidata@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata
--
Osma Suominen
D.Sc. (Tech), Information Systems Specialist
National Library of Finland
P.O. Box 26 (Kaikukatu 4)
00014 HELSINGIN YLIOPISTO
Tel. +358 50 3199529
osma.suomi...@helsinki.fi
http://www.nationallibrary.fi
_______________________________________________
Wikidata mailing list
Wikidata@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata
