On 17/09/12 18:55, Yaron Koren wrote: > Hi Markus, > > I'm curious about what you wrote about adding custom tables for > different properties - is there any documentation about it? > > As for the current code (at least in my extensions) that queries the SMW > database tables directly, here's an explanation, in order from least to > most challenging to "fix": > > - Semantic Internal Objects stores its own data in the tables; as you > note, this can probably be changed to use the new code for handlng > subobjects.
Yes. > > - Semantic Forms does direct querying in order to autocomplete on > substrings, in one function, > SFAutocompleteAPI::getAllValuesForProperty(). You can see it here: > > https://gerrit.wikimedia.org/r/gitweb?p=mediawiki/extensions/SemanticForms.git;a=blob;f=includes/SF_AutocompleteAPI.php > > There's nothing in the SMW API that allows for substring matching the > way SF does it (matching on the beginning of any word in the text); so > something would have to be added. Yes, this is tricky, and I don't think we can properly support this with this storage backend. Ideally, one would use Lucene for this (this is what Wikimedia does on its projects). What SMW does support in all storage functions is substring matching from the start of the string. Would this be a possible fallback option to retain some functionality while no other solution is found? Upcoming SMW versions should have hooks that allow extensions to listen to data changes. Through this, an extension could maintain its own tables (or other backend) to be used for special tasks such as auto completion. This would make this function independent of the storage backend, at the cost of using more storage. But different data access requirements should best be served by different data structures. > > - Semantic Drilldown does direct querying because it needs to generate > temporary tables, in order to improve performance: every time > Special:BrowseData is accessed, temporary tables are created that hold > the set of pages that match the current selected set of filters, and > then all the additional numbers and values on the pages are generated > off of those temp tables. To get that functionality into SMW would > require, I think, some serious additions to the code; and I don't know > if it would be worth the effort. Yes, this is tricky. Again, it might be better to maintain auxiliary data structures continuously based on watching SMW changes. Even with the temporary tables, SD is very slow right now due to the complicated queries it needs. Maybe one could reduce the effort by adding more (aggregate) information to the store that is updated on edit (relatively rare) instead of being updated on browsing (relatively frequent). > > (By the way, on that note - it would be great if SD could also handle > cases where an RDF triplestore is used to hold SMW's data, but I think > the lack of support for temporary tables in triplestores makes that > impossible, regardless of whether it's SD or SMW doing the work of > filtering. If anyone knows differently about that, though, please let me > know.) The current triple store support maintains all SQL tables as well. So the data can be taken from there in any case. The above approach (listen to changes to maintain your own tables) would also work in any case. But I agree that all of this is not so easy to realise and not something that can be done immediately. Making a fast faceted browser (SD 2.0 ;-) would be a project on its own. Markus > > -Yaron > > > On Mon, Sep 17, 2012 at 6:06 AM, Markus Krötzsch > <mar...@semantic-mediawiki.org <mailto:mar...@semantic-mediawiki.org>> > wrote: > > On 16/09/12 21:28, Yaron Koren wrote: > > Hi James, > > You make a valid point about the importance of SF and SD supporting > SQLStore3 in the near future. I plan to add support for the new > database > structure to those extensions, as well as to Semantic Internal > Objects, > which also has hardcoded calls to the SMW tables - ideally that can > happen in the next few weeks. I'm planning to do that by just adding > "if" statements to the relevant parts of the code, so that the > SQL used > changes depending on which SQL store is being used. From my > understanding of the new DB structure, it shouldn't be that hard > to do. > > > Yes and no. Regarding SIO you should be able to use SMW's API as > discussed in earlier emails, so no table-related issues there. For > SF, I am surprised that there is SQL-specific code at all. Some of > the current store API functions are only provided for the use by SF, > so if the kind of method that you need is not too hard to support, > we could maybe do the same there. > > SD will be more difficult to upgrade, I think (without knowing the > details of its code). One thing to know about the new store is that > it allows users to configure the tables that will be used. The > individual table layout is quite similar to what we already have, > but there can be a lot more (smaller) tables. In particular, some > (important) properties can get their own tables. > > Cheers, > > Markus > > > By the way, some people might argue that these extensions > shouldn't be > making direct SQL calls at all, which is a valid point; but the > problem > is that, as far as I know, there's no other way at the moment to > retrieve (or, in SIO's case, set) the necessary data. If someone > wants > to add such a capability to SMW for the use of one or more of these > extensions, I'd be glad to discuss it. > > -Yaron > > On Fri, Sep 14, 2012 at 10:45 AM, James HK > <jamesin.hongkon...@gmail.com <mailto:jamesin.hongkon...@gmail.com> > <mailto:jamesin.hongkong.1@__gmail.com > <mailto:jamesin.hongkon...@gmail.com>>> wrote: > > Hi, > > SMW 1.8beta has been released but some SMW core extensions > (SF, SD > etc.) using direct access to smw db tables that correspond > to the > SQLStore2 scheme[1]/[2] but are not compatible with the new > SQLStore3 > scheme. > > In case of SF 2.5 one can use SMW 1.8/SQLStore2 but not SMW > 1.8/SQLStore3 (because of the SQLStore2 specific db selects). > > How to resolve this situation that in both cases > (SQLStore2/SQLStore3) > the integration works without interference? > > I think this question needs some attention before releasing > SMW 1.8 > because when people start using the SQLStore3 in > combination with core > extensions like Semantic Forms or Semantic Drilldown > problems will > occur at the current implementation state. > > [1] > > https://gerrit.wikimedia.org/__r/gitweb?p=mediawiki/__extensions/SemanticForms.git;__a=blob;f=includes/SF___AutocompleteAPI.php;h=__8edc80fbefe0915a75451c2a9a6bef__a4f8033a14;hb=__b95c7db0e1ab010fdeb3e70ef0692b__8df406713a#l157 > > <https://gerrit.wikimedia.org/r/gitweb?p=mediawiki/extensions/SemanticForms.git;a=blob;f=includes/SF_AutocompleteAPI.php;h=8edc80fbefe0915a75451c2a9a6befa4f8033a14;hb=b95c7db0e1ab010fdeb3e70ef0692b8df406713a#l157> > > [2] > > https://gerrit.wikimedia.org/__r/gitweb?p=mediawiki/__extensions/SemanticDrilldown.__git;a=blob;f=specials/SD___BrowseData.php;h=__5f09df9f4259a598b6e23e86a9fc67__f6eb7027c9;hb=HEAD#l266 > > <https://gerrit.wikimedia.org/r/gitweb?p=mediawiki/extensions/SemanticDrilldown.git;a=blob;f=specials/SD_BrowseData.php;h=5f09df9f4259a598b6e23e86a9fc67f6eb7027c9;hb=HEAD#l266> > > Cheers > > > > ------------------------------__------------------------------__------------------ > Got visibility? > Most devs has no idea what their production app looks like. > Find out how fast your code is with AppDynamics Lite. > http://ad.doubleclick.net/clk;__262219671;13503038;y > <http://ad.doubleclick.net/clk;262219671;13503038;y>? > http://info.appdynamics.com/__FreeJavaPerformanceDownload.__html > <http://info.appdynamics.com/FreeJavaPerformanceDownload.html> > _________________________________________________ > Semediawiki-devel mailing list > Semediawiki-devel@lists.__sourceforge.net > <mailto:Semediawiki-devel@lists.sourceforge.net> > <mailto:Semediawiki-devel@__lists.sourceforge.net > <mailto:Semediawiki-devel@lists.sourceforge.net>> > > https://lists.sourceforge.net/__lists/listinfo/semediawiki-__devel > <https://lists.sourceforge.net/lists/listinfo/semediawiki-devel> > > > > > -- > WikiWorks · MediaWiki Consulting · http://wikiworks.com > > > > ------------------------------__------------------------------__------------------ > Everyone hates slow websites. So do we. > Make your web apps faster with AppDynamics > Download AppDynamics Lite for free today: > http://ad.doubleclick.net/clk;__258768047;13503038;j > <http://ad.doubleclick.net/clk;258768047;13503038;j>? > > http://info.appdynamics.com/__FreeJavaPerformanceDownload.__html > <http://info.appdynamics.com/FreeJavaPerformanceDownload.html> > > > > _________________________________________________ > Semediawiki-devel mailing list > Semediawiki-devel@lists.__sourceforge.net > <mailto:Semediawiki-devel@lists.sourceforge.net> > https://lists.sourceforge.net/__lists/listinfo/semediawiki-__devel > <https://lists.sourceforge.net/lists/listinfo/semediawiki-devel> > > > > > > -- > WikiWorks · MediaWiki Consulting · http://wikiworks.com ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ _______________________________________________ Semediawiki-devel mailing list Semediawiki-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/semediawiki-devel
