On 9/23/13 3:48 PM, Paul A. Houle wrote:
One of the goals of the infovore project is to develop something that targets this latency problem.https://github.com/paulhoule/infovore/wikiI’ve talked with a number of organizations that use DBpedia and Freebase data and almost all of them have either no solution or an incomplete solution for dealing with changes over time, something that’s absolutely necessary for sustainable social-semantic systems. Many of them have considered developing it but decided against developing it in house.
I bet they have :-)
When Freebase changed the format of the RDF dump I was able to adapt in less than a week (most of the time delay was that no official dump came out that week and I didn’t know what was going on); after fixing my code I was able to run against it interactively. Infovore is not using Hadoop so much for “big data”, but rather for “low latency”. Not extremely low latency, but once I trust the system enough it ought to have Freebase processed before I wake up on Sunday. The files are smaller than the official dump and will load faster, both things that will lower latency for the consumer. Right now the process is limited by the not-so-parallel process of ungzipping and re-gzipping the Freebase dump, but I believe a processing pipeline much more complex than the current one could still be run in less than a hour if you throw enough AWS instances at it The framework ought to work for any RDF data, including DBpedia (for which it has been tested), and I have a lot of stuff planned, including something that could “smush” Dbpedia identifiers to Freebase identifiers or the other way around to create a merged data set.
Nice!
Yes, what I am doing today is much simpler than what DBpedia is doing, but I’m taking a multi-pronged approach that focuses on process as much as technology. I’m keeping a notebook of how much time it takes me to do everything and learning how to squeeze out the errors and waste time with a battery of methods that are being documented.
Yes, that's the way to approach this matter. First pass, manual so you can get a good handle on the real time costs.
It is possible to run clusters in Amazon EMR by simply providing a credential pair – you don’t need to know much at all about AWS or Hadoop. I invite all of you to follow the this project and github and also follow the Google Group https://groups.google.com/forum/#!forum/infovore-basekb <https://groups.google.com/forum/#%21forum/infovore-basekb>
I am following it.
where you’ll get roughly two status reports a week and where people with questions get quick answers. I can definitely use contributions too, because the list of things I’d like to see are long and my own work will be focused on my own needs. Even if you don’t contribute, I welcome feature requests on the issue tracker.
This should be interesting to fellow DBpedia and LOD folk, for sure. Kingsley
*From:* Kingsley Idehen <mailto:kide...@openlinksw.com> *Sent:* Monday, September 23, 2013 1:37 PM*To:* dbpedia-discussion@lists.sourceforge.net <mailto:dbpedia-discussion@lists.sourceforge.net> *Subject:* Re: [Dbpedia-discussion] ANN: DBpedia 3.9 released, including wider infobox coverage, additional type statements, and new YAGO and Wikidata linksOn 9/23/13 1:00 PM, Tom Morris wrote:Congratulations on the new release!On Mon, Sep 23, 2013 at 6:27 AM, Christian Bizer <ch...@bizer.de <mailto:ch...@bizer.de>> wrote:1. the new release is based on updated Wikipedia dumps dating from March / April 2013 (the 3.8 release was based on dumps from June 2012), leading to an overall increase in the number of concepts in the English edition from 3.7 to 4.0 million things.What accounts for the long latency between the date of the dumps and the date of the release?TomA number of things: 1. Dataset QA -- the datasets are generated from mapping efforts 2. Dataset Loading & QA -- Linked Data Deployment (i.e., new URIs resolve to the new data) -- SPARQL Endpoint (new data is accessible via SPARQL endpoint) . Kingsley------------------------------------------------------------------------------ LIMITED TIME SALE - Full Year of Microsoft Training For Just $49.99! 1,500+ hours of tutorials including VisualStudio 2012, Windows 8, SharePoint 2013, SQL 2012, MVC 4, more. BEST VALUE: New Multi-Library Power Pack includes Mobile, Cloud, Java, and UX Design. Lowest price ever! Ends 9/20/13. http://pubads.g.doubleclick.net/gampad/clk?id=58041151&iu=/4140/ostg.clktrk _______________________________________________ Dbpedia-discussion mailing list Dbpedia-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dbpedia-discussion-- Regards, Kingsley Idehen Founder & CEO OpenLink Software Company Web:http://www.openlinksw.com Personal Weblog:http://www.openlinksw.com/blog/~kidehen Twitter/Identi.ca handle: @kidehen Google+ Profile:https://plus.google.com/112399767740508618350/about LinkedIn Profile:http://www.linkedin.com/in/kidehen ------------------------------------------------------------------------ ------------------------------------------------------------------------------ October Webinars: Code for Performance Free Intel webinars can help you accelerate application performance.Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most fromthe latest Intel processors and coprocessors. See abstracts and register >http://pubads.g.doubleclick.net/gampad/clk?id=60133471&iu=/4140/ostg.clktrk------------------------------------------------------------------------ _______________________________________________ Dbpedia-discussion mailing list Dbpedia-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dbpedia-discussion
-- Regards, Kingsley Idehen Founder & CEO OpenLink Software Company Web: http://www.openlinksw.com Personal Weblog: http://www.openlinksw.com/blog/~kidehen Twitter/Identi.ca handle: @kidehen Google+ Profile: https://plus.google.com/112399767740508618350/about LinkedIn Profile: http://www.linkedin.com/in/kidehen
smime.p7s
Description: S/MIME Cryptographic Signature
------------------------------------------------------------------------------ October Webinars: Code for Performance Free Intel webinars can help you accelerate application performance. Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from the latest Intel processors and coprocessors. See abstracts and register > http://pubads.g.doubleclick.net/gampad/clk?id=60133471&iu=/4140/ostg.clktrk
_______________________________________________ Dbpedia-discussion mailing list Dbpedia-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dbpedia-discussion
