Hi Gerard,
Query performance has never been this bad. Currently the lag is over 6
HOURS.. and rising
My previous question stands.. What is the plan because we do not cope.
how about each hosts their own? This would provide some relief.
Below, I attached a query to https://databus.dbpedia.org/repo/sparql to
query the latest download urls for the Wikidata-dbpedia extraction:
https://databus.dbpedia.org/dbpedia/wikidata
Here is the yasgui link: https://tinyurl.com/yy768vh3
We have a virtuoso docker image that takes the query, downloads the
files and fills a local sparql endpoint:
1. Download theDockerfile
https://github.com/dbpedia/dev.dbpedia.org/raw/master/pics/Dockerfile.dockerfile
2. Build|docker build -t databus-dump-triplestore .|
3. Load any Databus|?file|query:
|docker run -p 8890:8890 databus-dump-triplestore $(cat
file-with-query.sparql)|
Doing it this way would ease some load and the docker updates each week
and can be cronjobbed.
Note that this is for the Wikidata-DBpedia extraction:
http://svn.aksw.org/papers/2015/ISWC_Wikidata2DBpedia/public.pdf
Databus is an open platform, so as soon as Wikidata/WMF or somebody else
publishes the original wikidata dumps there, you can use the docker to
decentralise hosting.
All the best,
Sebastian
QUERY:
PREFIX dataid: <http://dataid.dbpedia.org/ns/core#>
PREFIX dataid-cv: <http://dataid.dbpedia.org/ns/cv#>
PREFIX dct: <http://purl.org/dc/terms/>
PREFIX dcat: <http://www.w3.org/ns/dcat#>
# Get all files
SELECT DISTINCT ?file WHERE {
?dataset dataid:artifact ?artifact .
FILTER (?artifact in (
<https://databus.dbpedia.org/dbpedia/wikidata/instance-types>,
<https://databus.dbpedia.org/dbpedia/wikidata/mappingbased-objects-uncleaned>,
<https://databus.dbpedia.org/dbpedia/wikidata/mappingbased-literals>,
<https://databus.dbpedia.org/dbpedia/wikidata/labels>,
<https://databus.dbpedia.org/dbpedia/wikidata/references>,
<https://databus.dbpedia.org/dbpedia/wikidata/ontology-subclassof>,
<https://databus.dbpedia.org/dbpedia/wikidata/sameas-external>,
<https://databus.dbpedia.org/dbpedia/wikidata/images>,
<https://databus.dbpedia.org/dbpedia/wikidata/geo-coordinates>,
<https://databus.dbpedia.org/dbpedia/wikidata/description>,
<https://databus.dbpedia.org/dbpedia/wikidata/mappingbased-properties-reified>,
<https://databus.dbpedia.org/dbpedia/wikidata/properties>,
<https://databus.dbpedia.org/dbpedia/wikidata/redirects>,
<https://databus.dbpedia.org/dbpedia/wikidata/sameas-all-wikis>,
<https://databus.dbpedia.org/dbpedia/wikidata/alias>
) ).
?dataset dcat:distribution ?distribution .
?dataset dct:hasVersion ?latestVersion .
{
SELECT (max(?version) as ?latestVersion) WHERE {
?dataset dataid:artifact ?artifact .
FILTER (?artifact in (
<https://databus.dbpedia.org/dbpedia/wikidata/instance-types>,
<https://databus.dbpedia.org/dbpedia/wikidata/mappingbased-objects-uncleaned>,
<https://databus.dbpedia.org/dbpedia/wikidata/mappingbased-literals>,
<https://databus.dbpedia.org/dbpedia/wikidata/labels>,
<https://databus.dbpedia.org/dbpedia/wikidata/references>,
<https://databus.dbpedia.org/dbpedia/wikidata/ontology-subclassof>,
<https://databus.dbpedia.org/dbpedia/wikidata/sameas-external>,
<https://databus.dbpedia.org/dbpedia/wikidata/images>,
<https://databus.dbpedia.org/dbpedia/wikidata/geo-coordinates>,
<https://databus.dbpedia.org/dbpedia/wikidata/description>,
<https://databus.dbpedia.org/dbpedia/wikidata/mappingbased-properties-reified>,
<https://databus.dbpedia.org/dbpedia/wikidata/properties>,
<https://databus.dbpedia.org/dbpedia/wikidata/redirects>,
<https://databus.dbpedia.org/dbpedia/wikidata/sameas-all-wikis>,
<https://databus.dbpedia.org/dbpedia/wikidata/alias>
) ).
?dataset dct:hasVersion ?version .
}
}
?distribution dcat:downloadURL ?file .
}
_______________________________________________
Wikidata mailing list
Wikidata@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata