Re: [Virtuoso-users] SPARQL Linked Data: howto configure sponger behind IRI deref for variables

[Jörn Hees]

On Tuesday 03 August 2010, Kingsley Idehen wrote:
> What are your Virtuoso INI settings? The behavior you describe basically 
> implies that cache handling is disabled.
> 
> The [SPARQL] section of the INI is what's relevant.

The section looks like this:

[SPARQL]
;ExternalQuerySource= 1
;ExternalXsltSource = 1
;DefaultGraph   = http://localhost:8890/dataspace
;ImmutableGraphs= http://localhost:8890/dataspace
ResultSetMaxRows   = 2
MaxQueryCostEstimationTime = 800; in seconds
MaxQueryExecutionTime  = 60 ; in seconds
DefaultQuery   = SELECT ?g count (*) WHERE {   GRAPH ?g { ?s ?p ?o 
} } GROUP BY ?g ORDER BY DESC 2 LIMIT 20
DeferInferenceRulesInit= 0  ; controls inference rules loading
;PingService= http://rpc.pingthesemanticweb.com/


Anything I should change?

Cheers,
Jörn

Re: [Virtuoso-users] SPARQL Linked Data: howto configure sponger behind IRI deref for variables

[Kingsley Idehen]

Jörn Hees wrote:

On Tuesday 03 August 2010, Kingsley Idehen wrote:
  

[...]
Jörn Hees wrote:


If I use:
  DEFINE get:proxy "localhost:8890"

with this query:
  DEFINE input:grab-all "yes"
  DEFINE input:grab-depth 1
  SELECT DISTINCT ?p ?lp ?o ?lo
  WHERE {
  
 ?p ?o.

OPTIONAL { ?p rdfs:label ?lp. }
OPTIONAL { ?o rdfs:label ?lo. }
   
   }


[...]

Cheers,
Jörn
  

All,

Have you digested this tutorial re. use of Sponger for de-referencing IRIs?

See:
http://virtuoso.openlinksw.com/presentations/SPARQL_Tutorials/SPARQL_Tutori
als_Part_7/SPARQL_Tutorials_Part_7.html



Yes, actually that's what lead me to the query above.
I looked all of the pragmas up in 
http://docs.openlinksw.com/virtuoso/rdfiridereferencing.html and now went 
through it again, but can't find how it helps with my problem.


What am I missing out?
  


I have the suspicion that cache handling is actually disabled via your 
INI due to default settings.


Kingsley

Cheers,
Jörn

--
The Palm PDK Hot Apps Program offers developers who use the
Plug-In Development Kit to bring their C/C++ apps to Palm for a share
of $1 Million in cash or HP Products. Visit us here for more details:
http://p.sf.net/sfu/dev2dev-palm
___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users
  



--

Regards,

Kingsley Idehen	  
President & CEO 
OpenLink Software 
Web: http://www.openlinksw.com

Weblog: http://www.openlinksw.com/blog/~kidehen
Twitter/Identi.ca: kidehen 

Re: [Virtuoso-users] SPARQL Linked Data: howto configure sponger behind IRI deref for variables

[Kingsley Idehen]

Jörn Hees wrote:

On Tuesday 03 August 2010, Ivan Mikhailov wrote:
  

What should be added to existing extension API configurable via
"define input:grab-resolver ..."
and "define input:grab-loader ..."
as they're described in
http://docs.openlinksw.com/virtuoso/rdfiridereferencing.html#rdfinputgrab



Sorry, saw your email too late.

Thanks for pointing met o input:grab-resolver, I think that's what I want...
Is there a HowTo for writing such an own grab-resolver method if you would 
want all IRIs to be resolved to your local store if they exist, use them; if 
not, fetch them and store them locally; not refetching anything more often 
than weekly? (searched your docs, but nothing found when I enter 
"RDF_GRAB_RESOLVER_DEFAULT"?!?)


Currently when calling the Proxy like this: 
http://localhost:8890/about/rdf/http/dbpedia.org/resource/Barack_Obama

it won't use the local DBpedia dump, as the graphs aren't "materialized":
I only have one graph from the DBpedia dump in the local store: 
"http://dbpedia.org";, all proxy-calls seem to introduce new graphs such as: 
"http://dbpedia.org/resource/Barack_Obama"; (and judging from the time they 
take it seems as if they don't use the local store) .

The same counts for all SPARQL-Queries doing a IRI-deref.

So would it be a good idea to substitute http://dbpedia.org with 
http://localhost:8890/ in the grab resolver?
Can I perhaps also do this for the proxy in total? (i.e., rewriting 
http://localhost:8890/about/rdf/http://dbpedia.org/resource/Barack_Obama to 
http://localhost:8890/resource/Barack_Obama )
  


What are your Virtuoso INI settings? The behavior you describe basically 
implies that cache handling is disabled.


The [SPARQL] section of the INI is what's relevant.

Kingsley

Cheers,
Jörn

--
The Palm PDK Hot Apps Program offers developers who use the
Plug-In Development Kit to bring their C/C++ apps to Palm for a share
of $1 Million in cash or HP Products. Visit us here for more details:
http://p.sf.net/sfu/dev2dev-palm
___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users
  



--

Regards,

Kingsley Idehen	  
President & CEO 
OpenLink Software 
Web: http://www.openlinksw.com

Weblog: http://www.openlinksw.com/blog/~kidehen
Twitter/Identi.ca: kidehen 

Re: [Virtuoso-users] SPARQL Linked Data: howto configure sponger behind IRI deref for variables

[Jörn Hees]

On Tuesday 03 August 2010, Kingsley Idehen wrote:
> [...]
> Jörn Hees wrote:
> > If I use:
> >   DEFINE get:proxy "localhost:8890"
> > 
> > with this query:
> >   DEFINE input:grab-all "yes"
> >   DEFINE input:grab-depth 1
> >   SELECT DISTINCT ?p ?lp ?o ?lo
> >   WHERE {
> >   
> >  ?p ?o.
> > OPTIONAL { ?p rdfs:label ?lp. }
> > OPTIONAL { ?o rdfs:label ?lo. }
> >
> >}
> > 
> > [...]
> > 
> > Cheers,
> > Jörn
> 
> All,
> 
> Have you digested this tutorial re. use of Sponger for de-referencing IRIs?
> 
> See:
> http://virtuoso.openlinksw.com/presentations/SPARQL_Tutorials/SPARQL_Tutori
> als_Part_7/SPARQL_Tutorials_Part_7.html

Yes, actually that's what lead me to the query above.
I looked all of the pragmas up in 
http://docs.openlinksw.com/virtuoso/rdfiridereferencing.html and now went 
through it again, but can't find how it helps with my problem.

What am I missing out?

Cheers,
Jörn

Re: [Virtuoso-users] SPARQL Linked Data: howto configure sponger behind IRI deref for variables

[Jörn Hees]

On Tuesday 03 August 2010, Ivan Mikhailov wrote:
> What should be added to existing extension API configurable via
> "define input:grab-resolver ..."
> and "define input:grab-loader ..."
> as they're described in
> http://docs.openlinksw.com/virtuoso/rdfiridereferencing.html#rdfinputgrab

Sorry, saw your email too late.

Thanks for pointing met o input:grab-resolver, I think that's what I want...
Is there a HowTo for writing such an own grab-resolver method if you would 
want all IRIs to be resolved to your local store if they exist, use them; if 
not, fetch them and store them locally; not refetching anything more often 
than weekly? (searched your docs, but nothing found when I enter 
"RDF_GRAB_RESOLVER_DEFAULT"?!?)

Currently when calling the Proxy like this: 
http://localhost:8890/about/rdf/http/dbpedia.org/resource/Barack_Obama
it won't use the local DBpedia dump, as the graphs aren't "materialized":
I only have one graph from the DBpedia dump in the local store: 
"http://dbpedia.org";, all proxy-calls seem to introduce new graphs such as: 
"http://dbpedia.org/resource/Barack_Obama"; (and judging from the time they 
take it seems as if they don't use the local store) .
The same counts for all SPARQL-Queries doing a IRI-deref.

So would it be a good idea to substitute http://dbpedia.org with 
http://localhost:8890/ in the grab resolver?
Can I perhaps also do this for the proxy in total? (i.e., rewriting 
http://localhost:8890/about/rdf/http://dbpedia.org/resource/Barack_Obama to 
http://localhost:8890/resource/Barack_Obama )

Cheers,
Jörn

Re: [Virtuoso-users] SPARQL Linked Data: howto configure sponger behind IRI deref for variables

[Kingsley Idehen]

Jörn Hees wrote:

On Tuesday 03 August 2010, Nathan wrote:
  

Aldo Bucchi wrote:


Hi Ivan, Jorn,

I've run into similar requirements. A catch-all solution to this could
be to surface the ability to specify a custom deref procedure

define input:iri_deref "DB.DBA.MY_CUSTOM_IRI_DEREF"

This should get all the context to decide whether to perform a fetch
and also have access to the sponger infrastructure.

This is a broad and raw idea, I know. The point is that the current
visibility into what happens under the covers is too low. Of course,
in an ideal world, sources should play along with the Linked Data
rules ( including specifying caching TTL ) but in reality you might
  

and ETag's!



want to override some sources, implement specific caching strategies,
etc.

Spongers provide some flexibility in the "how do I get stuff from a
IRI" part. But perhaps the controller part could also be "extensible".
  

FWIW, +1 unless you can simply designate sponger to use a proxy, which
would solve all use cases from what I can tell.



Yupp, I definitely agree. AFAIK there currently is no possibility to make use 
of let's say a local DBpedia mirror in case you're using the IRI-dereferencing 
options:


If I use:
  DEFINE get:proxy "localhost:8890"
with this query:
  DEFINE input:grab-all "yes"
  DEFINE input:grab-depth 1
  SELECT DISTINCT ?p ?lp ?o ?lo
  WHERE {
 ?p ?o.
OPTIONAL { ?p rdfs:label ?lp. }
OPTIONAL { ?o rdfs:label ?lo. }
   }
I'm told that get:proxy only makes sense with a FROM clause. So is there any 
possibility to use a proxy (e.g., the current datastore's URL rewriting) for 
IRI dereferencing, or perhaps even Aldo's suggestion of a special 
DB.DBA.MY_CUSTOM_IRI_DEREF ?


Cheers,
Jörn
  


All,

Have you digested this tutorial re. use of Sponger for de-referencing IRIs?


See: 
http://virtuoso.openlinksw.com/presentations/SPARQL_Tutorials/SPARQL_Tutorials_Part_7/SPARQL_Tutorials_Part_7.html



Kingsley


--
The Palm PDK Hot Apps Program offers developers who use the
Plug-In Development Kit to bring their C/C++ apps to Palm for a share
of $1 Million in cash or HP Products. Visit us here for more details:
http://p.sf.net/sfu/dev2dev-palm
___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users
  



--

Regards,

Kingsley Idehen	  
President & CEO 
OpenLink Software 
Web: http://www.openlinksw.com

Weblog: http://www.openlinksw.com/blog/~kidehen
Twitter/Identi.ca: kidehen 

Re: [Virtuoso-users] SPARQL Linked Data: howto configure sponger behind IRI deref for variables

[Jörn Hees]

On Tuesday 03 August 2010, Nathan wrote:
> Aldo Bucchi wrote:
> > Hi Ivan, Jorn,
> > 
> > I've run into similar requirements. A catch-all solution to this could
> > be to surface the ability to specify a custom deref procedure
> > 
> > define input:iri_deref "DB.DBA.MY_CUSTOM_IRI_DEREF"
> > 
> > This should get all the context to decide whether to perform a fetch
> > and also have access to the sponger infrastructure.
> > 
> > This is a broad and raw idea, I know. The point is that the current
> > visibility into what happens under the covers is too low. Of course,
> > in an ideal world, sources should play along with the Linked Data
> > rules ( including specifying caching TTL ) but in reality you might
> 
> and ETag's!
> 
> > want to override some sources, implement specific caching strategies,
> > etc.
> > 
> > Spongers provide some flexibility in the "how do I get stuff from a
> > IRI" part. But perhaps the controller part could also be "extensible".
> 
> FWIW, +1 unless you can simply designate sponger to use a proxy, which
> would solve all use cases from what I can tell.

Yupp, I definitely agree. AFAIK there currently is no possibility to make use 
of let's say a local DBpedia mirror in case you're using the IRI-dereferencing 
options:

If I use:
  DEFINE get:proxy "localhost:8890"
with this query:
  DEFINE input:grab-all "yes"
  DEFINE input:grab-depth 1
  SELECT DISTINCT ?p ?lp ?o ?lo
  WHERE {
 ?p ?o.
OPTIONAL { ?p rdfs:label ?lp. }
OPTIONAL { ?o rdfs:label ?lo. }
   }
I'm told that get:proxy only makes sense with a FROM clause. So is there any 
possibility to use a proxy (e.g., the current datastore's URL rewriting) for 
IRI dereferencing, or perhaps even Aldo's suggestion of a special 
DB.DBA.MY_CUSTOM_IRI_DEREF ?

Cheers,
Jörn

Re: [Virtuoso-users] SPARQL Linked Data: howto configure sponger behind IRI deref for variables

[Ivan Mikhailov]

Hello Aldo,

On Mon, 2010-08-02 at 20:36 -0400, Aldo Bucchi wrote:
> Hi Ivan, Jorn,
> 
> A catch-all solution to this could
> be to surface the ability to specify a custom deref procedure
> 
> define input:iri_deref "DB.DBA.MY_CUSTOM_IRI_DEREF"
> 
> This should get all the context to decide whether to perform a fetch
> and also have access to the sponger infrastructure.

What should be added to existing extension API configurable via
"define input:grab-resolver ..."
and "define input:grab-loader ..."
as they're described in
http://docs.openlinksw.com/virtuoso/rdfiridereferencing.html#rdfinputgrab
?

Best Regards,

Ivan Mikhailov
OpenLink Software
http://virtuoso.openlinksw.com

Re: [Virtuoso-users] SPARQL Linked Data: howto configure sponger behind IRI deref for variables

[Nathan]

Aldo Bucchi wrote:

Hi Ivan, Jorn,

I've run into similar requirements. A catch-all solution to this could
be to surface the ability to specify a custom deref procedure

define input:iri_deref "DB.DBA.MY_CUSTOM_IRI_DEREF"

This should get all the context to decide whether to perform a fetch
and also have access to the sponger infrastructure.

This is a broad and raw idea, I know. The point is that the current
visibility into what happens under the covers is too low. Of course,
in an ideal world, sources should play along with the Linked Data
rules ( including specifying caching TTL ) but in reality you might


and ETag's!


want to override some sources, implement specific caching strategies,
etc.

Spongers provide some flexibility in the "how do I get stuff from a
IRI" part. But perhaps the controller part could also be "extensible".


FWIW, +1 unless you can simply designate sponger to use a proxy, which 
would solve all use cases from what I can tell.


Best,

Nathan

Re: [Virtuoso-users] SPARQL Linked Data: howto configure sponger behind IRI deref for variables

[Aldo Bucchi]

Hi Ivan, Jorn,

I've run into similar requirements. A catch-all solution to this could
be to surface the ability to specify a custom deref procedure

define input:iri_deref "DB.DBA.MY_CUSTOM_IRI_DEREF"

This should get all the context to decide whether to perform a fetch
and also have access to the sponger infrastructure.

This is a broad and raw idea, I know. The point is that the current
visibility into what happens under the covers is too low. Of course,
in an ideal world, sources should play along with the Linked Data
rules ( including specifying caching TTL ) but in reality you might
want to override some sources, implement specific caching strategies,
etc.

Spongers provide some flexibility in the "how do I get stuff from a
IRI" part. But perhaps the controller part could also be "extensible".

As usual, the smartest way to go about this is wait until there are
enough use cases. I see virtuoso-users is picking up some speed ;)

Regards,
A





On Sat, Jul 31, 2010 at 12:18 AM, Ivan Mikhailov
 wrote:
> Hello Jörn,
>
>> is there any way to configure the sponger behind this to
>> 1. cache its lookups, so don't retrieve graphs already present in the local
>> quad store (the DEFINE get:soft "soft" seems to be what I want, but that only
>> works for graphs mentioned in a FROM or FROM NAMED clause),
>
> It caches things by default and control the expiration of cached copies
> by returned HTTP header lines. If a remote tells that the resource
> should not be cached or sets very small expiration time then the cache
> lifetime can not be lengthen. define get:refresh can force earlier
> reloading but can not delay it: it sets maximum allowed refresh time,
> not a strict value.
>
>> 2. do n-fetches in parallel and/or,
>
> It tries to fetch in parallel, via async_queue-s of "width" 8.
> Unfortunately that happens not in all cases.
>
>> 3. tell me if some object turned out not to be a linked data resource, but
>> just an URL (e.g., in the triple: (dbpedia:Barack_Obama dbpprop:reference
>> ) )
>
> I'll think what can be done. Right now there's nothing ready-to-use,
> because we follow Squids strategy and cache only good results, not
> failures. However it's possible to make sample RDF view to render data
> from table DB.DBA.SYS_HTTP_SPONGE and get access to status of currently
> cached copies.
>
> Best Regards,
>
> Ivan Mikhailov
> OpenLink Software
> http://virtuoso.openlinksw.com
>
>
>
> --
> The Palm PDK Hot Apps Program offers developers who use the
> Plug-In Development Kit to bring their C/C++ apps to Palm for a share
> of $1 Million in cash or HP Products. Visit us here for more details:
> http://p.sf.net/sfu/dev2dev-palm
> ___
> Virtuoso-users mailing list
> Virtuoso-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/virtuoso-users
>



-- 
Aldo Bucchi
@aldonline
skype:aldo.bucchi
http://aldobucchi.com/

PRIVILEGED AND CONFIDENTIAL INFORMATION
This message is only for the use of the individual or entity to which it is
addressed and may contain information that is privileged and confidential. If
you are not the intended recipient, please do not distribute or copy this
communication, by e-mail or otherwise. Instead, please notify us immediately by
return e-mail.

Re: [Virtuoso-users] SPARQL Linked Data: howto configure sponger behind IRI deref for variables

[Ivan Mikhailov]

Hello Jörn,

> is there any way to configure the sponger behind this to
> 1. cache its lookups, so don't retrieve graphs already present in the local 
> quad store (the DEFINE get:soft "soft" seems to be what I want, but that only 
> works for graphs mentioned in a FROM or FROM NAMED clause),

It caches things by default and control the expiration of cached copies
by returned HTTP header lines. If a remote tells that the resource
should not be cached or sets very small expiration time then the cache
lifetime can not be lengthen. define get:refresh can force earlier
reloading but can not delay it: it sets maximum allowed refresh time,
not a strict value.

> 2. do n-fetches in parallel and/or,

It tries to fetch in parallel, via async_queue-s of "width" 8.
Unfortunately that happens not in all cases.

> 3. tell me if some object turned out not to be a linked data resource, but 
> just an URL (e.g., in the triple: (dbpedia:Barack_Obama dbpprop:reference 
> ) )

I'll think what can be done. Right now there's nothing ready-to-use,
because we follow Squids strategy and cache only good results, not
failures. However it's possible to make sample RDF view to render data
from table DB.DBA.SYS_HTTP_SPONGE and get access to status of currently
cached copies.

Best Regards,

Ivan Mikhailov
OpenLink Software
http://virtuoso.openlinksw.com

[Virtuoso-users] SPARQL Linked Data: howto configure sponger behind IRI deref for variables

[Jörn Hees]

Hey all,

I'm currently working on a project where I want to ask a lot "cross-border" 
SPARQL-Queries like the following one:

SELECT DISTINCT ?p ?lp ?o ?lo
WHERE {
  ?p ?o.
 OPTIONAL { ?p rdfs:label ?lp. }
 OPTIONAL { ?o rdfs:label ?lo. }
}

(So I'd prefer a human readable version of potential URIs for ?p and ?o.)

Now virtuoso allows me to do so by using
DEFINE input:grab-all "yes"
DEFINE input:grab-depth 1

Nevertheless, as all drop-downs and tutorials say, the performance of this is 
very bad (takes around 15 minutes, when I execute the query).

So my question is: is there any way to configure the sponger behind this to
1. cache its lookups, so don't retrieve graphs already present in the local 
quad store (the DEFINE get:soft "soft" seems to be what I want, but that only 
works for graphs mentioned in a FROM or FROM NAMED clause),
2. do n-fetches in parallel and/or,
3. tell me if some object turned out not to be a linked data resource, but 
just an URL (e.g., in the triple: (dbpedia:Barack_Obama dbpprop:reference 
) )

Perhaps someone could point me into the right direction / has some helpful 
thoughts?

Cheers,
Jörn