Hi! > I am not sure how to evaluate this correctly. Scaling databases in > general is a "known hard problem" and graph databases a sub-field of it, > which are optimized for graph-like queries as opposed to column stores > or relational databases. If you say that "throwing hardware at the > problem" does not help, you are admitting that Blazegraph does not scale > for what is needed by Wikidata.
I think this is over-generalizing. We have a database that grew 10x over the last 4 years. We have certain hardware and software limits, both with existing hardware and in principle by hardware we could buy. We also have certain issues specific to graph databases that make scaling harder - for example, document databases, like ElasticSearch, and certain models of relational databases, shard easily. Sharding something like Wikidata graph is much harder, especially if the underlying database knows nothing about specifics of Wikidata data (which would be the case for all off-the-shelf databases). If we just randomly split the triples between several servers, we'd probably be just modeling a large but an extremely slow disk. So there needs to be some smarter solution, one that we'd unlike to develop inhouse but one that has already been verified by industry experience and other deployments. Is the issue specific to Blazegraph and can the issue be solved by switching platform? Maybe, we do not know yet. We do not have any better solution that guarantees us better scalability identified, but we have a plan on looking for that solution, given the resources. We also have a plan on improving the throughput of Blazegraph, which we're working on now. Non-sharding model might be hard to sustain indefinitely, but it is not clear it can't work in the short term, and also it is not clear that sharding model would deliver clear performance win, as it will have to involve network latencies inside the queries, which can significantly affect performance. This can only resolved by proper testing evaluation of the candidate solutions. > Then it is not a "cluster" in the sense of databases. It is more a > redundancy architecture like RAID 1. Is this really how BlazeGraph does I do not think our time here would be productively spent arguing semantics about what should and should not be called a "cluster". We call that setup a cluster, and I think now we all understand what we're talking about. > it? Don't they have a proper cluster solution, where they repartition > data across servers? Or is this independent servers a wikimedia staff > homebuild? If you mean sharded or replicated setup, as far as I know, Blazegraph does not support that (there's some support for replication IIRC but replication without sharding probably won't give us much improvement). We have a plan to evaluate a solution that does shard, given necessary resources. > Some info here: > > - We evaluated some stores according to their performance: > http://www.semantic-web-journal.net/content/evaluation-metadata-representations-rdf-stores-0 > > "Evaluation of Metadata Representations in RDF stores" Thanks for the link, it looks very interesting, I'll read it and see which parts we could use here. > - Virtuoso has proven quite useful. I don't want to advertise here, but > the thing they have going for DBpedia uses ridiculous hardware, i.e. > 64GB RAM and it is also the OS version, not the professional with > clustering and repartition capability. So we are playing the game since > ten years now: Everybody tries other databases, but then most people > come back to virtuoso. I have to admit that OpenLink is maintaining the > hosting for DBpedia themselves, so they know how to optimise. They > normally do large banks as customers with millions of write transactions > per hour. In LOD2 they also implemented column store features with > MonetDB and repartitioning in clusters. I do not know the details of your usage scenario, so before we get into comparisons, I'd like to understand: 1. Do your servers provide live synchronized updates with Wikdiata or DBPedia? How many updates per second that server can process? 2. How many queries per second this server is serving? What kind of queries are those? We did preliminary very limited evaluation of Virtuoso for hosting Wikidata, and it looks like it can load and host the necessary data (though it does not support some customizations we have now and we could not evaluate whether such customizations are possible) but it would require significant time investment to port all the functionality to it. Unfortunately, the lack of resources did not allow us to do fuller evaluation. Also, as I understand, "professional" capabilities of Virtuoso are closed-source and require paid license, which probably would be a problem to run it on WMF infrastructure unless we reach some kind of special arrangement. Since this arrangement will probably not include open-sourcing the enterprise part of Virtuoso, it should deliver a very significant, I dare say enormous advantage for us to consider running it in production. It may be possible that just OS version is also clearly superior to the point that it is worth migrating, but this needs to be established by evaluation. > - I recently heard a presentation from Arango-DB and they had a good > cluster concept as well, although I don't know anybody who tried it. The > slides seemed to make sense. We considered AgangoDB in the past, and it turned out we couldn't use it efficiently on the scales we need (could be our fault of course). They also use their own proprietary language for querying, which might be worth it if they deliver us a clear win on all other aspects, but that does not seem to be the case. Also, AgangoDB seems to be document database inside. This is not what our current data model is. While it is possible to model Wikidata in this way, again, changing the data model from RDF/SPARQL to a different one is an enormous shift, which can only be justified by an equally enormous improvement in some other areas, which currently is not clear. This project seems to be still very young. While I would be very interested if somebody took on themselves to model Wikidata in terms of ArangoDB documents, load the whole data and see what the resulting performance would be, I am not sure it would be wise for us to invest our team's - very limited currently - resources into that. Thanks, -- Stas Malyshev smalys...@wikimedia.org _______________________________________________ Wikidata mailing list Wikidata@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata
