Re: [Virtuoso-users] obtaining a copy/dump of an online SPARQL endpoint in a nice manner

2016-03-31 Thread Kingsley Idehen 
On 3/31/16 1:30 PM, Jörn Hees wrote:
> Hi,
>
> i developed some machine learning algorithms that i'd like to run against 
> various datasets.
> Many of them provide Virtuoso powered SPARQL endpoints online, but running my 
> algorithms against them would for sure not be considered "fair use".
>
> Some datasets provide dumps, so i'm able to play nice, load the dumps on a 
> local Virtuoso instance and torture that local instance with my algorithms.
>
> How can i do something similar in case there is no dump available for 
> download, but only a SPARQL endpoint?
>
> I was thinking about issuing a `construct where { ?s ?p ?o } limit X offset 
> Y` and stepping through the endpoint like that once, but the bigger the 
> offset, the slower the response time:
>
> http://dbpedia.org/sparql?default-graph-uri=http%3A%2F%2Fdbpedia.org=select+*+where+{%3Fs+%3Fp+%3Fo.}+limit+1+offset+40002=text%2Fhtml_redir_for_subjs=121_redir_for_hrefs==3=on
>
> Any suggestions how to improve this and do this in a "nice" way?
> Also maybe without the danger of skipping a lot of data by different orders?
>
> Best,
> Jörn

How are you arriving at data devoid or metadata about its origins?

You would be better served, ultimately, instantiating a dedicated
Virtuoso instance in the cloud for your specific needs. This instance
could load datasets from wherever, using some of the existing endpoints
(DBpedia and others) as a mechanism for exposing provenance data etc..

There is no nice way of trying to dump all the data from an existing
SPARQL endpoint.

Regards,

Kingsley Idehen   
Founder & CEO 
OpenLink Software 
Company Web: http://www.openlinksw.com
Personal Weblog 1: http://kidehen.blogspot.com
Personal Weblog 2: http://www.openlinksw.com/blog/~kidehen
Twitter Profile: https://twitter.com/kidehen
Google+ Profile: https://plus.google.com/+KingsleyIdehen/about
LinkedIn Profile: http://www.linkedin.com/in/kidehen
Personal WebID: http://kingsley.idehen.net/dataspace/person/kidehen#this




smime.p7s
Description: S/MIME Cryptographic Signature
--
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785471=/4140___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

Re: [Virtuoso-users] Temporary DB size and server restart

2016-03-31 Thread Ivan Mikhailov 
Hello Daniel,

It's practical to keep temp db file not erased after the server
shutdown, as a disk space reserve. In this case the file will be used
again at the next server run anyway, but the disk space will not be
occasionally occupied by other programs. It's impractical, however, to
keep surprisingly big reserve, esp. if the space allocated as a result
of an error that will never happen again. Say, a hash join of
"everything with everything" is not supposed to happen when the
application logic is correct, but it can create a terabyte-scale temp.db
if the computer is fast and the error is noticed only after hours of
run.

TempDBSize lets you chose the size of the file you permit to keep,
that's all. You can safely ignore this message if you have enough space
on the disk; otherwise adjust the TempDBSize to the size measured after
a usual correct run of the application server (multiplied by factor
1.5--2--3 for extra safety).

Best Regards,

Ivan Mikhailov
OpenLink Software
http://virtuoso.openlinksw.com

On Wed, 2016-03-30 at 16:08 +, Davis, Daniel (NIH/NLM) [C] wrote:
> I have a strange failure condition somewhere in my deployment.
> Virtuoso goes down, and whenever I restart it, I see something like
> this in the log:
> 
>  
> 
> Unlinked the temp db file /usr/nlm/virtuoso/db/virtuoso-temp.db as its
> size (8180MB) was greater than TempDBSize INI (10MB)
> 
>  
> 
> Now, that is a big difference.Why would the virtuoso-temp DB size
> get to be so much bigger than the limit?What removes it?
> 
>  
> 
> Dan Davis, Systems/Applications Architect (Contractor),
> 
> Office of Computer and Communications Systems,
> 
> National Library of Medicine, NIH




--
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785471=/4140
___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

Re: [Virtuoso-users] Temporary DB size and server restart

2016-03-31 Thread Davis, Daniel (NIH/NLM) [C] 
Virtuoso was terminated by the out-of-memory killer.   Most of my Virtuoso 
nodes had thread cleanup and resource cleanup set to 1, but not the one that 
failed.   So, this was a case of me not changing ThreadCleanupInterval and 
ResourceCleanupInterval when I installed this node.   I don’t trust my install 
script enough, and most people use puppet/chef or other tools.



From: Hugh Williams [mailto:hwilli...@openlinksw.com]
Sent: Wednesday, March 30, 2016 4:53 PM
To: Davis, Daniel (NIH/NLM) [C] 
Cc: virtuoso-users 
Subject: Re: [Virtuoso-users] Temporary DB size and server restart



Hi Daniel,



As indicated in the docs at 
http://docs.openlinksw.com/virtuoso/databaseadmsrv.html#ex_threadcleanupinterval
 :



*   TempDBSize

Controls the acceptable size of the temp database file. If on startup it's size 
(in MB) is greater than TempDBSize the file gets deleted and reset. This 
feature can be turned off by setting TempDBSize to 0. Note that the temp db 
file serves as an optimization storage only and doesn't have any client data 
that are not in either the main database files or the corresponding transaction 
log files.

If you Virtuoso database shutdown unexpectedly then the TempDB in use would not 
have been clean up which is what the message is indicated you should the 
consider manually removing as it contains no required data.



Thus you should focus on determining why the Virtuoso server shutdown 
unexpectedly, thus is anything of interest in the log prior to it shutting down 
or in the system log files that might indicate the cause of it shutdown ?



Best Regards

Hugh Williams

Professional Services

OpenLink Software, Inc.  //  http://www.openlinksw.com/

Weblog   -- http://www.openlinksw.com/blogs/

LinkedIn -- http://www.linkedin.com/company/openlink-software/

Twitter  -- http://twitter.com/OpenLink

Google+  -- http://plus.google.com/100570109519069333827/

Facebook -- http://www.facebook.com/OpenLinkSoftware

Universal Data Access, Integration, and Management Technology Providers







   On 30 Mar 2016, at 17:18, Davis, Daniel (NIH/NLM) [C] 
> wrote:



   It looks like this is merely a cache, and it’s OK for it to be bigger.It 
doesn’t look like it has anything to do with my Virtuoso shutdown issues.



   From: Davis, Daniel (NIH/NLM) [C]
   Sent: Wednesday, March 30, 2016 12:09 PM
   To: virtuoso-users 
>
   Subject: Temporary DB size and server restart



   I have a strange failure condition somewhere in my deployment.Virtuoso 
goes down, and whenever I restart it, I see something like this in the log:



   Unlinked the temp db file /usr/nlm/virtuoso/db/virtuoso-temp.db as its size 
(8180MB) was greater than TempDBSize INI (10MB)



   Now, that is a big difference.Why would the virtuoso-temp DB size get to 
be so much bigger than the limit?What removes it?



   Dan Davis, Systems/Applications Architect (Contractor),

   Office of Computer and Communications Systems,

   National Library of Medicine, NIH



   
--
   Transform Data into Opportunity.
   Accelerate data analysis in your applications with
   Intel Data Analytics Acceleration Library.
   Click to learn more.
   
http://pubads.g.doubleclick.net/gampad/clk?id=278785471=/4140___
   Virtuoso-users mailing list
   
Virtuoso-users@lists.sourceforge.net
   https://lists.sourceforge.net/lists/listinfo/virtuoso-users



--
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785471=/4140___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

Re: [Virtuoso-users] OAuth request_token endpoint returns OAuth Verification Failed: Bad Signature

2016-03-31 Thread Hugh Williams 
Hi Sunday,

The http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtOAuthSPARQL 
  
was pointing to and old ods framework vad and has been updated to point to the 
latest one for the 7.2.2 release. Note also that a different ods framework vad 
is require for open source and commercial as indicated on the page also. 

You can also build the required ods framework vad if compiling from open source 
archive by passing the --enable-ods-vad configure option …

Best Regards
Hugh Williams
Professional Services
OpenLink Software, Inc.  //  http://www.openlinksw.com/ 

Weblog   -- http://www.openlinksw.com/blogs/ 
LinkedIn -- http://www.linkedin.com/company/openlink-software/ 

Twitter  -- http://twitter.com/OpenLink 
Google+  -- http://plus.google.com/100570109519069333827/ 

Facebook -- http://www.facebook.com/OpenLinkSoftware 

Universal Data Access, Integration, and Management Technology Providers



> On 30 Mar 2016, at 21:29, Sunday Ayandokun  > wrote:
> 
> Hello,
> 
> I am having issue connecting to the OAuth to get token.
> 
> I installed the ods_framework_dav.vad package and generated SPARQL 
> application token and secret following the example here 
> http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtOAuthSPARQL 
> 
> 
> 
> I read this http://docs.openlinksw.com/virtuoso/voauth.html 
>  to understand how Oauth 
> request should be made to Virtuoso Oauth endpoint.
> 
> 
> http://localhost:8890/OAuth/request_token?oauth_version=1.0_nonce=72180780940317254641459358880_timestamp=1459299214_consumer_key=client_key_signature_method=HMAC-SHA1_signature=KmF8ngGHSBQJIDHTDw%2FWP%2Fdqgmc%3D
>  
> 
> 
> I am using a python library to sign the request.
> 
> client_key= token
> client_secret = secret
> 
> client = oauth1.Client(client_key, client_secret=client_secret)
> uri, headers, body = client.sign(request_token_url)
> 
> I am getting this response 'OAuth Verification Failed: Bad Signature'
> 
> 
> Kindly point me to the best approach.
> 
> 
> Kind Regards.
> Sunday Ayandokun
> University of Tartu
> Tel: +37256724770
> Skype: sunday.ayandokun
> 
> --
> Transform Data into Opportunity.
> Accelerate data analysis in your applications with
> Intel Data Analytics Acceleration Library.
> Click to learn more.
> http://pubads.g.doubleclick.net/gampad/clk?id=278785471=/4140___
>  
> 
> Virtuoso-users mailing list
> Virtuoso-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/virtuoso-users



smime.p7s
Description: S/MIME cryptographic signature
--
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785471=/4140___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users