[Making this into a new thread] Hi Krzysztof,
I was already wondering when I would hear from Wikia ... As you have noticed, running SMW and extensions on large sites (large in terms of content, or in terms of users) has special requirements. Typically, we suggest to use more conservative settings for querying, so that long and difficult queries do not occur. Similarly, some SMW extensions have not been developed for large sites, and can be problematic in their own right. But your users obviously want to keep the features that they already have, so we need to find better ways of addressing your problem. But first we need to separate concerns a little bit. You mention the following distinct problems: (1) Too many DB writes (about 60% in total) (2) Too many slow queries (about 90% from SMW) Moreover, your problem is not caused by SMW alone but by a number of SMW-related extensions. So there will be multiple issues that need addressing to fix this, and maybe even in multiple extensions. Let us first see how big the impact of the extensions you mention could be. Semantic Forms mainly leads to some additional reads (apparently no problem for you); the total number could possibly be reduced. It may also have some effect on query activity if certain autocompletion features are used. But otherwise I think it is unlikely to be the root of the problem. Semantic Drilldown might be more of a problem regarding complex queries. But it uses its own SQL queries, so it should be possible to find out how much of (2) comes from this extension. Semantic Drilldown should not contribute to (1). Are there any other extensions that use SMW on your site? Regarding SMW, I have some concrete ideas on what could be done for (1) and (2) but this will need more careful consideration first. I am grateful if you can help to track down the cause of the problem, but I am afraid that the changes in SMW core will still need to be done or at least reviewed carefully by myself -- which makes me kind of a bottleneck for the SMW part of your problem. I need to think about the required work a little further before I can promise anything. Regards, Markus On 22/02/2011 22:38, Krzysztof Krzyżaniak wrote: > I think it's would be right place to jump in. > > Hello, my name is Krzysztof Krzyżaniak a.k.a. eloy and I work for Wikia > Inc as backend team leader. We are probably (correct me if I am wrong) > on of the biggest user of Semantic Mediawiki suite. We currently have > enabled it on about 100 wikis for example on familypedia.wikia.com or > yugioh.wikia.com or www.wowwiki.com (but also on wikis which you > probably don't suspect for SMW interest like glee.wikia.com or > madmen.wikia.com). We would like to expand existence of SMW on Wikia > (for example lyrics would love it) but currently we cannot afford it > because of performance reasons. For example, our first cluster contains > about 30.000 wikis, mostly biggest ones. About 60% of writes in > databases came from SMW extensions (SemanticMediawiki, > SemanticDrilldown, SemanticForms), also about 90% queries from slow logs > are from SMW. > > I am here to find a way for scaling SMW on our wikis. But also I think > that it will be benefit for every SMW user because we want to help > improve SMW. > > What you can expect: > - "real world" cases, actually lot of them :) > - bugs :) (filled in bugzilla of course) > - bug fixes and patches (either as diff or direct svn commits if you > prefer that way) > - questions > > We can offer engineering hours and testbeds. > > For a start I have question for Roadmap: SMW light - how complete it is? > What's missing? When you expect it will be ready? How can we help? > > eloy > ------------------------------------------------------------------------------ Free Software Download: Index, Search & Analyze Logs and other IT data in Real-Time with Splunk. Collect, index and harness all the fast moving IT data generated by your applications, servers and devices whether physical, virtual or in the cloud. Deliver compliance at lower cost and gain new business insights. http://p.sf.net/sfu/splunk-dev2dev _______________________________________________ Semediawiki-devel mailing list Semediawiki-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/semediawiki-devel
