I'm with David in Camp 3, but there are also some things i'd like to bring
up for discussion while the group is considering a new store:
- job queue, smw_refresh to store properties. If you have millions of
articles, with each article template laden with semantic properties, there
are scalability and maintenance issues. Especially if you use bots to pump
a lot of data. Is this a mediawiki constraint or can we store properties
near-real time? FYI, in the CKAN project, they have a datastorer plugin (
https://github.com/okfn/ckanext-datastorer using celeryproject.org) that
parses structured data asynchronously so it can be queried via an API
- Historical semantic data. Would be nice if we can query historical data
as pages are updated over time, i.e. including a date range when doing
queries to show how properties change over time.
- having additional metadata/provenance info. Apart from when, who/what
made the assertion? This could well become a semantic system catalog/data
dictionary of sorts that can be used to compute semantic statistics as well
as optimize queries.
- Wikidata integration. If I'm not mistaken, there was talk of WikiData
and SMW ultimately "joining" together in the indeterminate future.
Perhaps, this could be the start of that process as some of these issues
may have already been considered by the WikiData team.
Thanks,
Joel
=======================================================
Think Different! (http://en.wikipedia.org/wiki/Think_different#Text)
Imagine Different! (http://www.youtube.com/watch?v=H5tOgRD4EqY)
On Wed, Oct 16, 2013 at 10:13 AM, david mason <
vid_semediawiki-de...@zooid.org> wrote:
>
> With regard to ES and data recovery/transactions, if SMW continues to be
> able to generate this data at any time it doesn't seem to be much of an
> issue. ES is also horizontally scalable as one of its main features, and
> supports geo features and advanced search, although graph traversal is
> manual and commits are near-real-time.
>
> I am mainly proposing this for the simplicity of the operators. Asking
> them to set up, for one SMW instance, MW, MySQL, SMW, ES for MW search at
> least, and one or two additional stores seems like a lot.
>
> I would guess that there are three kinds of SMW users; 1. those happy
> using it as a flexible self-contained front end built on MW for forms and
> pages, 2. those who would like to use it for Semantic Web / LOD type
> purposes (formal ontology design, enforcement, inference, and shared data
> between sites using web standards), and 3. those who would at least like a
> solid option/path to 2.
>
> For the many members of the community who would benefit from a real focus
> on an RDF store and schema support, I would clearly support something like
> Richard's "stack," but it might add a lot of complexity to hosting and
> development. Probably many SMW users now are using inexpensive hosting
> plans which wouldn't support this broader stack, and as I understand it the
> current SMW PHP API is not cleanly designed up so it may basically be a
> reinvention (which could be a good thing but would be disruptive).
>
> For myself I work in a mix of applications and am in solidly in camp 3 as
> a way forward, fwiw.
>
> And I can't help but wonder how WikiData fits into the mix. (=
>
> David
>
>
>
>
>
> On 16 October 2013 09:48, Richard Banks <richard.bank...@gmail.com> wrote:
>
>> Hi,
>>
>> Just to add to the conversation, I would also recommend ElasticSearch as
>> a great solution for the search side of things. There are also cases of
>> people using it as the sole data store. However, I believe caution should
>> be taken against such an approach since ES currently doesn't provide much
>> in the way of data recovery or transactions.
>>
>> For this reason, ES is typically deployed in combination with a data
>> storage technology that does support these factors, such as Mongo. ES
>> allows you to define what's known as "rivers", and these pull data out of a
>> configured data source and into the index, thus providing the benefits of
>> its powerful search (which is literally insane).
>>
>> In terms of making use of the rich inherent graph structure of the data
>> at the higher level, a GraphDB would make sense as suggested by Joel. One
>> GraphDB that might be worth a look is Titan, which has been developed by
>> the Tinkerpop guys I believe. Its a distributed graph database which also
>> (interestingly) supports ElasticSearch. It also abstracts over many data
>> stores/formats (including RDF) out-of-the-box. ES is a clever move IMO
>> because one of the challenges in graph search is jumping into the graph in
>> the first place, and it looks like they use the ES index to do this.
>>
>> So, you could almost just use Titan for search, get all the benefits of
>> graph traversals etc., and have it manage your ES index too.
>>
>> Regards,
>> Richard
>>
>>
>> ------------------------------------------------------------------------------
>> October Webinars: Code for Performance
>> Free Intel webinars can help you accelerate application performance.
>> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most
>> from
>> the latest Intel processors and coprocessors. See abstracts and register >
>>
>> http://pubads.g.doubleclick.net/gampad/clk?id=60135031&iu=/4140/ostg.clktrk
>> _______________________________________________
>> Semediawiki-devel mailing list
>> Semediawiki-devel@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/semediawiki-devel
>>
>>
>
>
> ------------------------------------------------------------------------------
> October Webinars: Code for Performance
> Free Intel webinars can help you accelerate application performance.
> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most
> from
> the latest Intel processors and coprocessors. See abstracts and register >
> http://pubads.g.doubleclick.net/gampad/clk?id=60135031&iu=/4140/ostg.clktrk
> _______________________________________________
> Semediawiki-devel mailing list
> Semediawiki-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/semediawiki-devel
>
>
------------------------------------------------------------------------------
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from
the latest Intel processors and coprocessors. See abstracts and register >
http://pubads.g.doubleclick.net/gampad/clk?id=60135031&iu=/4140/ostg.clktrk
_______________________________________________
Semediawiki-devel mailing list
Semediawiki-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/semediawiki-devel
