[Virtuoso-users] Big Linked Data Benchmarking Survey ...

[Hugh Williams]

Hi All,

The HOBBIT project [1] aims at abolishing the barriers in the adoption and 
deployment of Big Linked Data in organisations, by means of open benchmarking 
reports that allow them to assess the fitness of existing solutions for their 
purposes. These benchmarks are based on data that reflects reality and measures 
industry-relevant Key Performance Indicators (KPIs) with comparable results 
using standardized hardware.

Are you working on a solution in the Linked Data Lifecycle, do you require a 
Linked Data solution or are you innovating in the Linked Data Lifecycle ? If 
so, we invite you to answer some questions regarding benchmarking and potential 
use cases. The survey should not take more than five minutes to fill in the 
online survey at [2] ...

[1] http://project-hobbit.eu/
[2] http://goo.gl/forms/1iRIoG4Xpb

Best Regards
Hugh Williams
Professional Services
OpenLink Software, Inc.  //  http://www.openlinksw.com/
Weblog   -- http://www.openlinksw.com/blogs/
LinkedIn -- http://www.linkedin.com/company/openlink-software/
Twitter  -- http://twitter.com/OpenLink
Google+  -- http://plus.google.com/100570109519069333827/
Facebook -- http://www.facebook.com/OpenLinkSoftware
Universal Data Access, Integration, and Management Technology Providers





smime.p7s
Description: S/MIME cryptographic signature
--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

Re: [Virtuoso-users] Virtuoso User Accounts Limitation

[Sunday Ayandokun]

Hello Hugh, 

Thanks for your response.

I am sorry for the early conclusion.

I have an sql file with about 100,000 records to insert new user.

The first time I tried using this: isql-vt  dba dba 'EXEC=status()’ 
VirtuosoUserCreateQuery.sql

The query was successful. And when I did select count(*) from sys_users;, I got 
3050.

I tried this for about 3 times same result, definitely am missing something.

Kindly point me to the best approach to creating multiples users.


Thanks
Sunday

> On Feb 28, 2016, at 11:24 PM, Hugh Williams  wrote:
> 
> Hi Sunday,
> 
> How have you concluded there is a 3050 user limit, as we have system in-house 
> with 80K+ users ?
> 
> SQL> select count(*) from sys_users;
> count
> INTEGER
> ___
> 
> 88016
> 
> 1 Rows. -- 72 msec.
> SQL>
> 
> Best Regards
> Hugh Williams
> Professional Services
> OpenLink Software, Inc.  //  http://www.openlinksw.com/ 
> 
> Weblog   -- http://www.openlinksw.com/blogs/ 
> 
> LinkedIn -- http://www.linkedin.com/company/openlink-software/ 
> 
> Twitter  -- http://twitter.com/OpenLink 
> Google+  -- http://plus.google.com/100570109519069333827/ 
> 
> Facebook -- http://www.facebook.com/OpenLinkSoftware 
> 
> Universal Data Access, Integration, and Management Technology Providers
> 
> 
> 
>> On 28 Feb 2016, at 17:37, Sunday Ayandokun > > wrote:
>> 
>> Hello all,
>> 
>> I am working on my Master Thesis, as part of my experimentation I have a Use 
>> Case that requires creating about 100,000 users on the Virtuoso conductor.
>> 
>> I found out that, I can only create 3050 users. 
>> 
>> My questions:
>> 
>> 1. Can this limit be increased?
>> 2. If yes, where can I do that.
>> 
>> 
>> Kind regards.
>> Sunday 
>> --
>> Site24x7 APM Insight: Get Deep Visibility into Application Performance
>> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
>> Monitor end-to-end web transactions and take corrective actions now
>> Troubleshoot faster and improve end-user experience. Signup Now!
>> http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 
>> 
>> ___
>> Virtuoso-users mailing list
>> Virtuoso-users@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/virtuoso-users
> 

--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

Re: [Virtuoso-users] Virtuoso User Accounts Limitation

[Hugh Williams]

Hi Sunday,

How have you concluded there is a 3050 user limit, as we have system in-house 
with 80K+ users ?

SQL> select count(*) from sys_users;
count
INTEGER
___

88016

1 Rows. -- 72 msec.
SQL>

Best Regards
Hugh Williams
Professional Services
OpenLink Software, Inc.  //  http://www.openlinksw.com/
Weblog   -- http://www.openlinksw.com/blogs/
LinkedIn -- http://www.linkedin.com/company/openlink-software/
Twitter  -- http://twitter.com/OpenLink
Google+  -- http://plus.google.com/100570109519069333827/
Facebook -- http://www.facebook.com/OpenLinkSoftware
Universal Data Access, Integration, and Management Technology Providers



> On 28 Feb 2016, at 17:37, Sunday Ayandokun  wrote:
> 
> Hello all,
> 
> I am working on my Master Thesis, as part of my experimentation I have a Use 
> Case that requires creating about 100,000 users on the Virtuoso conductor.
> 
> I found out that, I can only create 3050 users. 
> 
> My questions:
> 
> 1. Can this limit be increased?
> 2. If yes, where can I do that.
> 
> 
> Kind regards.
> Sunday 
> --
> Site24x7 APM Insight: Get Deep Visibility into Application Performance
> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
> Monitor end-to-end web transactions and take corrective actions now
> Troubleshoot faster and improve end-user experience. Signup Now!
> http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
> ___
> Virtuoso-users mailing list
> Virtuoso-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/virtuoso-users



smime.p7s
Description: S/MIME cryptographic signature
--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

Re: [Virtuoso-users] infrequent errors on parallel querying

[Hugh Williams]

Hi Andreas,

Generally there should be no create additional indexes as the 2 Full & 3 
partial indexes have been found to be sufficient for most use cases, with only 
one use case encountered where an addtional partial index was required as 
detailed at:


http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtRDFPerformanceTuning#Index%20Scheme%20Selection

Thus what indexes (STATISTICS DB.DBA.RDF_QUAD;) are created on these various 
instances you have are they all the same or do they have different indexes as 
the more indexes the more large the database and hence more memory required for 
hosting it. Also are the triple counts the same on all these instances ?

Did you make any of the other INI file changes in my previous email, certainly 
“AdjustVectorSize” should be set to 0 ?

You should also consider profiling the queries to determine if the best quey 
plan is being used, which can be done with the “profile()” function as detailed 
at:


http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtTipsAndTricksAanalyzingSPARQLQuery
http://docs.openlinksw.com/virtuoso/databaseadmsrv.html#querylogging  - 
Query Logging 

Note there is are also some INI file params that can be set to control query 
optimisation ie plans as if a bad plan is being chosen then these options can 
in some cases enable better plans to be chosen, see:


http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtQueryOptDiagnostic

If you can provide query plans, database statistics then these can be analysed 
to trying and determine the cause of long running queries …

It certainly would be interesting to see how the query plans and database stats 
vary being the instance where the query runs in msecs and the other were it 
runs in 40+mins ...

Best Regards
Hugh Williams
Professional Services
OpenLink Software, Inc.  //  http://www.openlinksw.com/
Weblog   -- http://www.openlinksw.com/blogs/
LinkedIn -- http://www.linkedin.com/company/openlink-software/
Twitter  -- http://twitter.com/OpenLink
Google+  -- http://plus.google.com/100570109519069333827/
Facebook -- http://www.facebook.com/OpenLinkSoftware
Universal Data Access, Integration, and Management Technology Providers



> On 28 Feb 2016, at 14:58, Nolle, Andreas  wrote:
> 
> Dear Hugh,
> 
> many thanks for your reply!
> Sorry, I forgot to mentioned that I increased up to 24 core and 228 GB memory.
> 
> I'm still on trying to optimize the performance and to avoid transaction 
> deadlocks... but unfortunately without any success so far...
> 
> I've already tried to create some additional indexes, like PSOG, but the 
> evaluation of some filter parts take in some cases really long (more than 40 
> minutes)...
> 
> For me it is also very strange why the evaluation of query
> 
> define input:default-graph-uri 
> PREFIX xsd:  
> PREFIX owl: 
> PREFIX quest:   
> PREFIX rdf: 
> PREFIX rdfs:
> 
> SELECT ?x ?b
> WHERE {
> ?x  ?b .
> FILTER ( ?x >=  ) .
> FILTER ( ?x <  ) .
> }
> 
> at instance running at port 8895 will take only some ms, but executing query
> 
> define input:default-graph-uri  
> PREFIX xsd:  
> PREFIX owl: 
> PREFIX quest:   
> PREFIX rdf: 
> PREFIX rdfs:
> 
> SELECT ?x ?b
> WHERE {
>  SERVICE  {
> ?x  ?b .
> FILTER ( ?x >=  ) .
> FILTER ( ?x <  ) .
>  } .
> }
> 
> at another instance is running possibly indefinitely... (stopped the 
> evaluation after 45 minutes).
> 
> Best
> Andy
> 
> 
> 
> -Ursprüngliche Nachricht-
> Von: Hugh Williams [mailto:hwilli...@openlinksw.com] 
> Gesendet: Sonntag, 28. Februar 2016 04:14
> An: Nolle, Andreas 
> Cc: virtuoso-users@lists.sourceforge.net
> Betreff: Re: [Virtuoso-users] infrequent errors on parallel querying
> 
> Hi Andreas,
> 
> In you INI file “MaxClientConnections” and “ServerThreads” are the same thing 
> the latter being the original name which is retained for compatibility 
> reasons. Why do you set these params to 1000 on both sections of the INI file 
> ? I would reduce them to some thing like 200 as especially the HTTP Server 
> ones are pre-allocated on server startup which will consume significant 
> server resources for these threads.
> 
> Looking at the Vectored execution INI file params you h

[Virtuoso-users] Virtuoso User Accounts Limitation

[Sunday Ayandokun]

Hello all,

I am working on my Master Thesis, as part of my experimentation I have a Use 
Case that requires creating about 100,000 users on the Virtuoso conductor.

I found out that, I can only create 3050 users. 

My questions:

1. Can this limit be increased?
2. If yes, where can I do that.


Kind regards.
Sunday 
--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

Re: [Virtuoso-users] infrequent errors on parallel querying

[Nolle, Andreas]

Dear Hugh,

many thanks for your reply!
Sorry, I forgot to mentioned that I increased up to 24 core and 228 GB memory.

I'm still on trying to optimize the performance and to avoid transaction 
deadlocks... but unfortunately without any success so far...

I've already tried to create some additional indexes, like PSOG, but the 
evaluation of some filter parts take in some cases really long (more than 40 
minutes)...

For me it is also very strange why the evaluation of query

define input:default-graph-uri 
PREFIX xsd:  
PREFIX owl: 
PREFIX quest:   
PREFIX rdf: 
PREFIX rdfs:

SELECT ?x ?b
WHERE {
  ?x  ?b .
  FILTER ( ?x >=  ) .
  FILTER ( ?x <  ) .
}

at instance running at port 8895 will take only some ms, but executing query

define input:default-graph-uri  
PREFIX xsd:  
PREFIX owl: 
PREFIX quest:   
PREFIX rdf: 
PREFIX rdfs:

SELECT ?x ?b
WHERE {
   SERVICE  {
  ?x  ?b .
  FILTER ( ?x >=  ) .
  FILTER ( ?x <  ) .
   } .
}

at another instance is running possibly indefinitely... (stopped the evaluation 
after 45 minutes).

Best
Andy



-Ursprüngliche Nachricht-
Von: Hugh Williams [mailto:hwilli...@openlinksw.com] 
Gesendet: Sonntag, 28. Februar 2016 04:14
An: Nolle, Andreas 
Cc: virtuoso-users@lists.sourceforge.net
Betreff: Re: [Virtuoso-users] infrequent errors on parallel querying

Hi Andreas,

In you INI file “MaxClientConnections” and “ServerThreads” are the same thing 
the latter being the original name which is retained for compatibility reasons. 
Why do you set these params to 1000 on both sections of the INI file ? I would 
reduce them to some thing like 200 as especially the HTTP Server ones are 
pre-allocated on server startup which will consume significant server resources 
for these threads.

Looking at the Vectored execution INI file params you have set:

MaxQueryMem  = 32G  ; memory allocated to query processor
VectorSize   = 1000 ; initial parallel query vector (array of query 
operations) size
MaxVectorSize= 100  ; query vector size threshold.
AdjustVectorSize = 1
ThreadsPerQuery  = 24

Why is MaxQueryMem set so high ie 32G as this will all consume significant 
system memory when multiple queries are being run, thus I would set it to 
something like the 2G default to start off with. How much memory is available 
on the system ?

AdjustVectorSize should be set to 0 in most use cases …

I assume you machine has 24 cores which is why you set "ThreadsPerQuery 
 = 24” ?

See the following documentation on vectored query execution:

http://docs.openlinksw.com/virtuoso/databaseadmsrv.html#confvectexec - 
Configuring Vectored Execution

http://docs.openlinksw.com/virtuoso/databaseadmsrv.html#tunparamsmworkload  - 
Tuning Parameters for Multiuser Workloads

and the following on Transaction deadlocks and how to determine and avoid them:

http://docs.openlinksw.com/virtuoso/databaseadmsrv.html#perfdiag - 
Performance diagnostics 

http://docs.openlinksw.com/virtuoso/databaseadmsrv.html#TRANSACTION_ISOLATION_LEVELS
 - Transaction Metrics, Diagnostics and Optimization

Best Regards
Hugh Williams
Professional Services
OpenLink Software, Inc.  //  http://www.openlinksw.com/
Weblog   -- http://www.openlinksw.com/blogs/
LinkedIn -- http://www.linkedin.com/company/openlink-software/
Twitter  -- http://twitter.com/OpenLink
Google+  -- http://plus.google.com/100570109519069333827/
Facebook -- http://www.facebook.com/OpenLinkSoftware
Universal Data Access, Integration, and Management Technology Providers



> On 26 Feb 2016, at 08:03, Nolle, Andreas  wrote:
> 
> The critical error is now that after some query evaluations I get a lot of 
> "Virtuoso 40001 Error SR172: Transaction deadlocked" errors. For me this is 
> really strange because I've set the DefaultIsolation = 2 and only SELECT 
> queries are executed. Please notice that I've slightly changed some 
> parameters in the ini files.
> 
> It would be really nice if you could help me on that and give me some 
> suggestions how transaction errors can be avoided on evaluating SELECT 
> queries. Please find the current log and ini files of each Virtuoso instance 
> as well as the corresp