Re: [SMW-devel] SMW performance
Resources are cheap only when projects are not popular - when they get popular, resources become very expensive. Yeh, Video editing spoils people ;) Sergey On Dec 30, 2007 4:58 AM, cnit <[EMAIL PROTECTED]> wrote: > > Yes, but just throwing resources at something doesn't make it efficient. > Resources are cheap. Even major server brands such as HP have moved their > production to China. > > OK, I am doing videoediting among other things, that's why I am for > hardware resources. > > > As for code cache/accelerator, I agree - improvement is quite big > because MW > > is doing lots of includes plus MW can use it for cache as well - I use > APC > > and it works pretty good. > Of course. > Dmitriy > > - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/___ Semediawiki-devel mailing list Semediawiki-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/semediawiki-devel
Re: [SMW-devel] SMW Performance
On Sonntag, 30. Dezember 2007, Sergey Chernyshev wrote:
> Hmm. I didn't realize there is a way to remove $smwgQDefaultNamespaces
> restriction and this will enable all namespaces instead of disabling them.
>
> Why is it that this setting not set to NULL by default then? I don't see
> any point in restricting namespaces unless it's absolutely necessary for
> security reasons or something.
I agree. Done.
Markus
>
> Sergey
>
> On Dec 29, 2007 10:29 AM, Markus Krötzsch <[EMAIL PROTECTED]> wrote:
> > On Freitag, 28. Dezember 2007, Lau, William (NIH/CIT) [E] wrote:
> > > We have a set of semantic queries in a template. That template is used
> > > in some pages. However, by looking at the database process list, it
> > > seems that those set of queries are processed whenever a page is
> > > requested, even when the template is not used by the requested page
> > > (e.g. special pages).
> >
> > Do I understand this correctly? The delivery of special pages that are
> > completely unrelated to said template triggers the ask-queries contained
> > therein? This would be very strange behaviour indeed (I cannot currently
> > imagine how or why this should happen in MediaWiki)!
> >
> > > All the SQL queries are generated by the
> > > getQueryResult function. Since those queries are very computational
> > > intensive, this "bug" slows down the entire site. If we take the inline
> > > queries out of the template or change $smwgQEnabled to false, the site
> > > becomes fast again. Has anyone experienced the same issue?
> >
> > In general, if queries on some site are too slow, it is useful to
> > configure
> > SMW to support faster querying (with less features, of course). Basic
> > settings one can try to speed up querying are:
> >
> > include_once('extensions/SemanticMediaWiki/includes/SMW_Settings.php');
> > $smwgQSubcategoryDepth = 0;
> > $smwgQSubpropertyDepth = 0;
> > $smwgQEqualitySupport = SMW_EQ_NONE;
> > $smwgQDefaultNamespaces = NULL;
> > enableSemantics("semedia-wiki.localhost");
> >
> > Those settings will speed up basically all queries, disabling all support
> > for
> > property and category hierarchies, equality (redirects), and namespace
> > restrictions (i.e. queries consider pages in all namespaces, including,
> > e.g.,
> > User:). You can experiment which of those, if any, affects your query
> > performance positively.
> >
> > If you have problems with too complex user-generated queries, then the
> > parameters $smwgQMaxSize and $smwgQMaxDepth are an option to restrict
> > this.
> >
> > In general, it should be emphasised that queries should be used in a
> > targetted
> > way. Ontoworld.org had the infamous template {{ask}} for some time, which
> > included queries for almost anything, which would just not appear if no
> > results would be obtained. Most wikis should rather have single query
> > templates for special purposes instead of trying to have one for all.
> >
> > Anyway, for further optimisation, we need some pointer to your site, or
> > at least some statistical information concerning its size
> > (Special:SemanticStatistics) and the query structure. Did you mention the
> > SMW
> > version you use? Some of the above assume SMW1.0-RC3, and none will work
> > prior to SMW1.0-RC1.
> >
> > Markus
> >
> >
> > --
> > Markus Krötzsch
> > Institut AIFB, Universät Karlsruhe (TH), 76128 Karlsruhe
> > phone +49 (0)721 608 7362fax +49 (0)721 608 5998
> > [EMAIL PROTECTED]www http://korrekt.org
> >
> > -
> > This SF.net email is sponsored by: Microsoft
> > Defy all challenges. Microsoft(R) Visual Studio 2005.
> > http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/
> > ___
> > Semediawiki-devel mailing list
> > Semediawiki-devel@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/semediawiki-devel
--
Markus Krötzsch
Institut AIFB, Universät Karlsruhe (TH), 76128 Karlsruhe
phone +49 (0)721 608 7362fax +49 (0)721 608 5998
[EMAIL PROTECTED]www http://korrekt.org
signature.asc
Description: This is a digitally signed message part.
-
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/___
Semediawiki-devel mailing list
Semediawiki-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/semediawiki-devel
Re: [SMW-devel] SMW performance
> Yes, but just throwing resources at something doesn't make it efficient. Resources are cheap. Even major server brands such as HP have moved their production to China. OK, I am doing videoediting among other things, that's why I am for hardware resources. > As for code cache/accelerator, I agree - improvement is quite big because MW > is doing lots of includes plus MW can use it for cache as well - I use APC > and it works pretty good. Of course. Dmitriy - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Semediawiki-devel mailing list Semediawiki-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/semediawiki-devel
Re: [SMW-devel] SMW Performance
Hmm. I didn't realize there is a way to remove $smwgQDefaultNamespaces
restriction and this will enable all namespaces instead of disabling them.
Why is it that this setting not set to NULL by default then? I don't see any
point in restricting namespaces unless it's absolutely necessary for
security reasons or something.
Sergey
On Dec 29, 2007 10:29 AM, Markus Krötzsch <[EMAIL PROTECTED]> wrote:
> On Freitag, 28. Dezember 2007, Lau, William (NIH/CIT) [E] wrote:
> > We have a set of semantic queries in a template. That template is used
> > in some pages. However, by looking at the database process list, it
> > seems that those set of queries are processed whenever a page is
> > requested, even when the template is not used by the requested page
> > (e.g. special pages).
>
> Do I understand this correctly? The delivery of special pages that are
> completely unrelated to said template triggers the ask-queries contained
> therein? This would be very strange behaviour indeed (I cannot currently
> imagine how or why this should happen in MediaWiki)!
>
> > All the SQL queries are generated by the
> > getQueryResult function. Since those queries are very computational
> > intensive, this "bug" slows down the entire site. If we take the inline
> > queries out of the template or change $smwgQEnabled to false, the site
> > becomes fast again. Has anyone experienced the same issue?
>
> In general, if queries on some site are too slow, it is useful to
> configure
> SMW to support faster querying (with less features, of course). Basic
> settings one can try to speed up querying are:
>
> include_once('extensions/SemanticMediaWiki/includes/SMW_Settings.php');
> $smwgQSubcategoryDepth = 0;
> $smwgQSubpropertyDepth = 0;
> $smwgQEqualitySupport = SMW_EQ_NONE;
> $smwgQDefaultNamespaces = NULL;
> enableSemantics("semedia-wiki.localhost");
>
> Those settings will speed up basically all queries, disabling all support
> for
> property and category hierarchies, equality (redirects), and namespace
> restrictions (i.e. queries consider pages in all namespaces, including,
> e.g.,
> User:). You can experiment which of those, if any, affects your query
> performance positively.
>
> If you have problems with too complex user-generated queries, then the
> parameters $smwgQMaxSize and $smwgQMaxDepth are an option to restrict
> this.
>
> In general, it should be emphasised that queries should be used in a
> targetted
> way. Ontoworld.org had the infamous template {{ask}} for some time, which
> included queries for almost anything, which would just not appear if no
> results would be obtained. Most wikis should rather have single query
> templates for special purposes instead of trying to have one for all.
>
> Anyway, for further optimisation, we need some pointer to your site, or at
> least some statistical information concerning its size
> (Special:SemanticStatistics) and the query structure. Did you mention the
> SMW
> version you use? Some of the above assume SMW1.0-RC3, and none will work
> prior to SMW1.0-RC1.
>
> Markus
>
>
> --
> Markus Krötzsch
> Institut AIFB, Universät Karlsruhe (TH), 76128 Karlsruhe
> phone +49 (0)721 608 7362fax +49 (0)721 608 5998
> [EMAIL PROTECTED]www http://korrekt.org
>
> -
> This SF.net email is sponsored by: Microsoft
> Defy all challenges. Microsoft(R) Visual Studio 2005.
> http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/
> ___
> Semediawiki-devel mailing list
> Semediawiki-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/semediawiki-devel
>
>
--
Sergey Chernyshev
http://www.sergeychernyshev.com/
-
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/___
Semediawiki-devel mailing list
Semediawiki-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/semediawiki-devel
Re: [SMW-devel] SMW performance
Yes, but just throwing resources at something doesn't make it efficient. As for code cache/accelerator, I agree - improvement is quite big because MW is doing lots of includes plus MW can use it for cache as well - I use APC and it works pretty good. Sergey On Dec 28, 2007 6:36 AM, cnit <[EMAIL PROTECTED]> wrote: > > Forget my previous post. The problem goesaway when I removed one > > template. It seems the performance issue is related tothe > > application instead of the database. > Try setting up eAccelerator for PHP, maybe it would help a bit. Also, > I believe that MW/SMW requires dedicated server (co-location). We've > tried usual low-cost hosting (with hundreds of other's virtual hosts) > and even MW alone was crawling.. > Also, it was slow under Windows. It's ok with Linux server. Of course > you could also try dedicated MySQL server. MW even supports MySQL > clustering and web load-balancing, because it's being used by > wikipedia - one of the busiest and largest sites in the world. > Dmitriy > - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/___ Semediawiki-devel mailing list Semediawiki-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/semediawiki-devel
Re: [SMW-devel] SMW Performance
On Freitag, 28. Dezember 2007, Lau, William (NIH/CIT) [E] wrote:
> We have a set of semantic queries in a template. That template is used
> in some pages. However, by looking at the database process list, it
> seems that those set of queries are processed whenever a page is
> requested, even when the template is not used by the requested page
> (e.g. special pages).
Do I understand this correctly? The delivery of special pages that are
completely unrelated to said template triggers the ask-queries contained
therein? This would be very strange behaviour indeed (I cannot currently
imagine how or why this should happen in MediaWiki)!
> All the SQL queries are generated by the
> getQueryResult function. Since those queries are very computational
> intensive, this "bug" slows down the entire site. If we take the inline
> queries out of the template or change $smwgQEnabled to false, the site
> becomes fast again. Has anyone experienced the same issue?
In general, if queries on some site are too slow, it is useful to configure
SMW to support faster querying (with less features, of course). Basic
settings one can try to speed up querying are:
include_once('extensions/SemanticMediaWiki/includes/SMW_Settings.php');
$smwgQSubcategoryDepth = 0;
$smwgQSubpropertyDepth = 0;
$smwgQEqualitySupport = SMW_EQ_NONE;
$smwgQDefaultNamespaces = NULL;
enableSemantics("semedia-wiki.localhost");
Those settings will speed up basically all queries, disabling all support for
property and category hierarchies, equality (redirects), and namespace
restrictions (i.e. queries consider pages in all namespaces, including, e.g.,
User:). You can experiment which of those, if any, affects your query
performance positively.
If you have problems with too complex user-generated queries, then the
parameters $smwgQMaxSize and $smwgQMaxDepth are an option to restrict this.
In general, it should be emphasised that queries should be used in a targetted
way. Ontoworld.org had the infamous template {{ask}} for some time, which
included queries for almost anything, which would just not appear if no
results would be obtained. Most wikis should rather have single query
templates for special purposes instead of trying to have one for all.
Anyway, for further optimisation, we need some pointer to your site, or at
least some statistical information concerning its size
(Special:SemanticStatistics) and the query structure. Did you mention the SMW
version you use? Some of the above assume SMW1.0-RC3, and none will work
prior to SMW1.0-RC1.
Markus
--
Markus Krötzsch
Institut AIFB, Universät Karlsruhe (TH), 76128 Karlsruhe
phone +49 (0)721 608 7362fax +49 (0)721 608 5998
[EMAIL PROTECTED]www http://korrekt.org
signature.asc
Description: This is a digitally signed message part.
-
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/___
Semediawiki-devel mailing list
Semediawiki-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/semediawiki-devel
Re: [SMW-devel] SMW Performance
> We have a set of semantic queries in a template. Thattemplate is > used in some pages. However, by looking at the database processlist, > it seems that those set of queries are processed whenever a page > isrequested, even when the template is not used by the requested > page (e.g.special pages). All the SQL queries are generated by the > getQueryResultfunction. Since those queries are very computational > intensive, this“bug” slows down the entire site. If we take the > inline queries outof the template or change $smwgQEnabled to false, > the site becomes fast again.Has anyone experienced the same issue? Perhaps the query is too complex or data set is too big. I believe that "false" conditions of templates are executed by MW's parser in every case (it's documented behavior of MW's parser) - just the output of false conditions would be dropped (not being shown). Markus probably considers using better caching mechanisms for ask queries, however it's not implemented yet. Maybe you could try running MW behind squid accelerator, I've seen it mentioned somewhere at http://mediawiki.org Also, maybe you should move MySQL database to dedicated server. Dmitriy - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Semediawiki-devel mailing list Semediawiki-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/semediawiki-devel
Re: [SMW-devel] SMW performance
> Forget my previous post. The problem goesaway when I removed one > template. It seems the performance issue is related tothe > application instead of the database. Try setting up eAccelerator for PHP, maybe it would help a bit. Also, I believe that MW/SMW requires dedicated server (co-location). We've tried usual low-cost hosting (with hundreds of other's virtual hosts) and even MW alone was crawling.. Also, it was slow under Windows. It's ok with Linux server. Of course you could also try dedicated MySQL server. MW even supports MySQL clustering and web load-balancing, because it's being used by wikipedia - one of the busiest and largest sites in the world. Dmitriy - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Semediawiki-devel mailing list Semediawiki-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/semediawiki-devel
Re: [SMW-devel] SMW performance
Forget my previous post. The problem goes away when I removed one template. It seems the performance issue is related to the application instead of the database. From: Wang, Alex (NIH/CIT) [E] Sent: Thursday, December 27, 2007 4:46 PM To: semediawiki-devel@lists.sourceforge.net Subject: [SMW-devel] SMW performance It is very slow (30 seconds) to open any pages including "Main Page". Checking the processes in mysqld found the following sql statements for each page respectively. I guess `prop11`, `prop8`, `cats6` are temporary tables, which make it hard to tune these statements. Does anyone know if these statements look alright? Your help are greatly appreciated. SELECT DISTINCT `page`.page_title as title, `page`.page_namespace as namespace, `page`.page_id as id FROM `prop11`, `prop8`, `cats6`, `page` INNER JOIN `categorylinks` AS cl5 ON cl5.cl_from=`page`.page_id INNER JOIN `smw_relations` AS rel7 ON rel7.subject_id=`page`.page_id LEFT JOIN `redirect` AS rd9 ON rd9.rd_from=rel7.object_id INNER JOIN `smw_relations` AS rel10 ON rel10.subject_id=`page`.page_id LEFT JOIN `redirect` AS rd12 ON rd12.rd_from=rel10.object_id WHERE ((`page`.page_namespace='0') OR (`page`.page_namespace='6')) AND ((cats6.title=cl5.cl_to) AND (prop8.title=rel7.relation_title AND ((rel7.object_title='Main_Page' AND rel7.object_namespace=0) OR (rd9.rd_title='Main_Page' AND rd9.rd_namespace=0))) AND (prop11.title=rel10.relation_title AND ((rel10.object_title='Parotid' AND rel10.object_namespace=0) OR (rd12.rd_title='Parotid' AND rd12.rd_namespace=0 LIMIT 51 SELECT DISTINCT `page`.page_title as title, `page`.page_namespace as namespace, `page`.page_id as id FROM `prop11`, `prop8`, `cats6`, `page` INNER JOIN `categorylinks` AS cl5 ON cl5.cl_from=`page`.page_id INNER JOIN `smw_relations` AS rel7 ON rel7.subject_id=`page`.page_id LEFT JOIN `redirect` AS rd9 ON rd9.rd_from=rel7.object_id INNER JOIN `smw_relations` AS rel10 ON rel10.subject_id=`page`.page_id LEFT JOIN `redirect` AS rd12 ON rd12.rd_from=rel10.object_id WHERE ((`page`.page_namespace='0') OR (`page`.page_namespace='6')) AND ((cats6.title=cl5.cl_to) AND (prop8.title=rel7.relation_title AND ((rel7.object_title='Monobook.css' AND rel7.object_namespace=0) OR (rd9.rd_title='Monobook.css' AND rd9.rd_namespace=0))) AND (prop11.title=rel10.relation_title AND ((rel10.object_title='Parotid' AND rel10.object_namespace=0) OR (rd12.rd_title='Parotid' AND rd12.rd_namespace=0 LIMIT 51 SELECT DISTINCT `page`.page_title as title, `page`.page_namespace as namespace, `page`.page_id as id FROM `prop11`, `prop8`, `cats6`, `page` INNER JOIN `categorylinks` AS cl5 ON cl5.cl_from=`page`.page_id INNER JOIN `smw_relations` AS rel7 ON rel7.subject_id=`page`.page_id LEFT JOIN `redirect` AS rd9 ON rd9.rd_from=rel7.object_id INNER JOIN `smw_relations` AS rel10 ON rel10.subject_id=`page`.page_id LEFT JOIN `redirect` AS rd12 ON rd12.rd_from=rel10.object_id WHERE ((`page`.page_namespace='0') OR (`page`.page_namespace='6')) AND ((cats6.title=cl5.cl_to) AND (prop8.title=rel7.relation_title AND ((rel7.object_title='Protein_Clusters' AND rel7.object_namespace=0) OR (rd9.rd_title='Protein_Clusters' AND rd9.rd_namespace=0))) AND (prop11.title=rel10.relation_title AND ((rel10.object_title='Parotid' AND rel10.object_namespace=0) OR (rd12.rd_title='Parotid' AND rd12.rd_namespace=0 LIMIT 51 SELECT DISTINCT `page`.page_title as title, `page`.page_namespace as namespace, `page`.page_id as id FROM `prop11`, `prop8`, `cats6`, `page` INNER JOIN `categorylinks` AS cl5 ON cl5.cl_from=`page`.page_id INNER JOIN `smw_relations` AS rel7 ON rel7.subject_id=`page`.page_id LEFT JOIN `redirect` AS rd9 ON rd9.rd_from=rel7.object_id INNER JOIN `smw_relations` AS rel10 ON rel10.subject_id=`page`.page_id LEFT JOIN `redirect` AS rd12 ON rd12.rd_from=rel10.object_id WHERE ((`page`.page_namespace='0') OR (`page`.page_namespace='6')) AND ((cats6.title=cl5.cl_to) AND (prop8.title=rel7.relation_title AND ((rel7.object_title='HSPP_Cluster:0067' AND rel7.object_namespace=0) OR (rd9.rd_title='HSPP_Cluster:0067' AND rd9.rd_namespace=0))) AND (prop11.title=rel10.relation_title AND ((rel10.object_title='Parotid' AND rel10.object_namespace=0) OR (rd12.rd_title='Parotid' AND rd12.rd_namespace=0 LIMIT 51 - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/___ Semediawiki-devel mailing list Semediawiki-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/semediawiki-devel
