Hi Hugh.
Actually, what you say makes sense and fits in my case. As you probably
remember I was trying to run a transaction that needed days to execute.
Trying to speed things up and confirm if everything was OK, I restarted the
VM numerous times. Obviously, this could have caused some inconsistency in
the virtuoso database.
Despite this error, virtuoso seems to be running normally now. In the mean
time I have bulk loaded a big set of triples without error. Moreover, there
is no "/var/log/messages" file and no error.
The output of *backup ‘/dev/null’; *seems normal to me:
*Done. -- 6890738 msec.*
Thank you very much for your answer.
Kind regards,
Pantelis Natsiavas
PS Please note that the link you sent me points to a rather general manual
and does not explain anything specific regarding integrity checking.
2016-07-17 2:51 GMT+03:00 Hugh Williams <hwilli...@openlinksw.com>:
> Hi Pantelis,
>
> The error in the log snippet you provided:
>
> 19:49:11 It is impossible to have a database file
> /media/VirtuosoDBDrive/virtuoso.db with a length not multiple of 2MB.
> 19:49:11 The process must have last terminated while growing the file.
> 19:49:11 Please contact OpenLink Customer Support
>
> implies that Virtuoso probably crashed or shutdown unexpectedly when
> Virtuoso was in the middle of increasing the database size by 2MB segment.
> Although is seems to the start successfully as indicated in the log, but
> should run a database integrity check to ensure there is no corruption of
> the database by running the following from isql:
>
> backup ‘/dev/null’;
>
> See,
> http://docs.openlinksw.com/virtuoso/databaseadmsrv.html#diagnosingrepairing
>
> Also, are any reports in the system log ie /var/log/messages to indicate a
> problem with Virtuoso ?
>
> Best Regards
> Hugh Williams
> Professional Services
> OpenLink Software, Inc. // http://www.openlinksw.com/
> Weblog -- http://www.openlinksw.com/blogs/
> LinkedIn -- http://www.linkedin.com/company/openlink-software/
> Twitter -- http://twitter.com/OpenLink
> Google+ -- http://plus.google.com/100570109519069333827/
> Facebook -- http://www.facebook.com/OpenLinkSoftware
> Universal Data Access, Integration, and Management Technology Providers
>
> On 14 Jul 2016, at 07:54, Pantelis Natsiavas <natsia...@gmail.com> wrote:
>
> Hi Hugh.
>
> The part of interest in the log file is currently attached. Sorry for not
> attaching it in my previous email. I have reduced only to the log messages
> that could be of interest, in the specific time span, around "Please
> contact OpenLink Customer Support" message.
>
> Yes I have changed the swappiness parameter to vm.swappiness = 10. I have
> not yet managed to use the debugger as the above error occurs. I will not
> insist on that.
>
> The status(); response has been copied and pasted as it came out in the
> isql-v command line.
>
> Actually, what happened in my last two tries executing the above, is that
> the SPARQL query executed (partially) as isql-v stopped the transaction
> because the "connection with the server had been lost" (or something like
> that).
>
> Thank you once more for your time.
>
> Kind regards,
> Pantelis Natsiavas
>
> 2016-07-12 14:30 GMT+03:00 Hugh Williams <hwilli...@openlinksw.com>:
>
>> Hi Pantelis,
>>
>> I don’t see the indicated attachment to your email with the virtuoso log
>> ?
>>
>> Is that status(); command output complete as it normally should the CPU
>> % being consume by the server when running and apart from the status
>> command, I don’t see any other processes indicated as running ?
>>
>> Note you can use the Virtuoso PL Debugger to see what
>> procedures/statements are running on the server continuously as follows:
>>
>> $ isql 1111 dba <pwd> -D
>> OpenLink Interactive PL Debugger (Virtuoso).
>> Type EXIT to exit, HELP for help
>> Debug session estabilished to 1111
>> DEBUG> info threads
>> @1111:4 in DB.DBA.test_proc1 () at 11
>> DEBUG>
>>
>> See the following documentation of additional details on usage:
>>
>> http://docs.openlinksw.com/virtuoso/pldebugger.html#pldebugger
>>
>>
>> Also, do you have the Linux swappiness parameter set to minimise
>> swapping to disk which will degrade performance as detailed at:
>>
>>
>> http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtRDFPerformanceTuning#Linux-only%20--%20”swappiness
>> <http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtRDFPerformanceTuning#Linux-only%20--%20%E2%80%9Dswappiness>
>> ”
>>
>> Best Regards
>> Hugh Williams
>> Professional Services
>> OpenLink Software, Inc. // http://www.openlinksw.com/
>> Weblog -- http://www.openlinksw.com/blogs/
>> LinkedIn -- http://www.linkedin.com/company/openlink-software/
>> Twitter -- http://twitter.com/OpenLink
>> Google+ -- http://plus.google.com/100570109519069333827/
>> Facebook -- http://www.facebook.com/OpenLinkSoftware
>> Universal Data Access, Integration, and Management Technology Providers
>>
>> On 7 Jul 2016, at 15:46, Pantelis Natsiavas <natsia...@gmail.com> wrote:
>>
>> Thank you Hugh for your answer.
>>
>> Virtuoso runs on a VM with 32GBytes of RAM. This VM is dedicated to the
>> specific virtuoso instance and hosted in cloud infrastructure.
>>
>> I have left my transaction to run until Tuesday (about 4 days !!!) and
>> still had no response. The VM reported 95% usage of memory but still no
>> response. The response in the status() command was the following:
>> *OpenLink Virtuoso Server*
>> *Version 07.20.3214-pthreads for Linux as of Oct 14 2015 *
>> *Started on: 2016-07-02 16:13 GMT+3*
>>
>> *Database Status:*
>> * File size 0, 10407936 pages, 3793820 free.*
>> * 2720000 buffers, 130665 used, 4 dirty 0 wired down, repl age 0 0 w. io
>> 0 w/crsr.*
>> * Disk Usage: 131941 reads avg 0 msec, 0% r 0% w last 0 s, 31533 writes
>> flush 4.255 MB,*
>> * 562 read ahead, batch = 224. Autocompact 0 in 0 out, 0% saved.*
>> *Gate: 388 2nd in reads, 0 gate write waits, 0 in while read 0 busy
>> scrap. *
>> *Log = /usr/local/var/lib/virtuoso/db/virtuoso.trx, 4146 bytes*
>> *6613693 pages have been changed since last backup (in checkpoint state)*
>> *Current backup timestamp: 0x0000-0x00-0x00*
>> *Last backup date: unknown*
>> *Clients: 1 connects, max 1 concurrent*
>> *RPC: 8 calls, 1 pending, 1 max until now, 0 queued, 0 burst reads (0%),
>> 0 second 0M large, 22M max*
>> *Checkpoint Remap 0 pages, 0 mapped back. 35 s atomic time.*
>> * DB master 10407936 total 3793820 free 0 remap 0 mapped back*
>> * temp 256 total 251 free*
>>
>> *Lock Status: 0 deadlocks of which 0 2r1w, 39 waits,*
>> * Currently 1 threads running 0 threads waiting 0 threads in vdb.*
>> *Pending:*
>>
>> *Client 1111:1:-3: Account: dba, 215 bytes in, 289 bytes out, 1 stmts.*
>> *PID: 30445, OS: unix, Application: unknown, IP#: 127.0.0.1*
>> *Transaction status: PENDING, 1 threads.*
>> *Locks: *
>>
>>
>> *Running Statements:*
>> * Time (msec) Text*
>> * 567 status()*
>>
>>
>> *Hash indexes*
>>
>>
>> On Wednesday, I restarted the VM in order to make sure that the
>> transaction was cleared, and executed the query through the isql-v while
>> enabling log. The usage of memory is currently about 60% and I got plenty
>> of log messages that I cannot really interpret. I attach the log file in
>> the current email. Perhaps, you could provide me with some guidance on what
>> I should do, based on the log file.
>>
>> Please note the following:
>>
>> The values in the virtuoso.ini configuration file are:
>>
>> *NumberOfBuffers = 2720000*
>> *MaxDirtyBuffers = 2000000*
>>
>> The response in the status(); command in the SQL is the following:
>>
>> *OpenLink Virtuoso Server*
>> *Version 07.20.3214-pthreads for Linux as of Oct 14 2015 *
>> *Started on: 2016-07-06 20:17 GMT+3*
>>
>> *Database Status:*
>> * File size 0, 11027968 pages, 3947693 free.*
>> * 2720000 buffers, 483125 used, 31 dirty 0 wired down, repl age 0 0 w.
>> io 0 w/crsr.*
>> * Disk Usage: 472040 reads avg 0 msec, 0% r 0% w last 0 s, 936646
>> writes flush 10.35 MB,*
>> * 2533 read ahead, batch = 168. Autocompact 530282 in 446316 out, 15%
>> saved col ac: 45742 in 12% saved.*
>> *Gate: 446 2nd in reads, 0 gate write waits, 0 in while read 0 busy
>> scrap. *
>> *Log = /usr/local/var/lib/virtuoso/db/virtuoso.trx, 1403 bytes*
>> *7079218 pages have been changed since last backup (in checkpoint state)*
>> *Current backup timestamp: 0x0000-0x00-0x00*
>> *Last backup date: unknown*
>> *Clients: 1 connects, max 1 concurrent*
>> *RPC: 6 calls, 1 pending, 1 max until now, 0 queued, 0 burst reads (0%),
>> 0 second 0M large, 152M max*
>> *Checkpoint Remap 589 pages, 0 mapped back. 464 s atomic time.*
>> * DB master 11027968 total 3947693 free 589 remap 2 mapped back*
>> * temp 256 total 251 free*
>>
>> *Lock Status: 0 deadlocks of which 0 2r1w, 20 waits,*
>> * Currently 1 threads running 0 threads waiting 0 threads in vdb.*
>> *Pending:*
>>
>> *Client 1111:1:-2: Account: dba, 215 bytes in, 289 bytes out, 1 stmts.*
>> *PID: 10273, OS: unix, Application: unknown, IP#: 127.0.0.1*
>> *Transaction status: PENDING, 1 threads.*
>> *Locks: *
>>
>>
>> *Running Statements:*
>> * Time (msec) Text*
>> * 673 status()*
>>
>>
>> *Hash indexes*
>>
>> Thank you in advance for your help.
>>
>> Kind regards,
>> Pantelis Natsiavas
>>
>>
>>
>>
>> 2016-07-02 23:22 GMT+03:00 Hugh Williams <hwilli...@openlinksw.com>:
>>
>>> Hi Pantelis,
>>>
>>> SPARUL operations are performed as part of a transaction, which is
>>> stored in memory until the operation is completed and committed to the
>>> database, and for such a large graph would consume a significant amount
>>> system memory, possibly all available memory and not being able to
>>> complete. Thus how much memory is available on the system and how much is
>>> allocated for Virtuoso data buffers ie NumberOfBuffers INI file param and
>>> how much memory is left ? You can run the “status();” command from isql to
>>> see the Buffers setting and other Virtuoso server status metrics.
>>>
>>> You can also try prepending the query with the "DEFINE sql:log-enable 3”
>>> pragma which sets transaction log mode to autocommit during the insert
>>> significantly reducing the memory requirement which may then allow the
>>> operation to complete, as is required when deleting large graphs as
>>> documented at:
>>>
>>>
>>> http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtTipsAndTricksGuideDeleteLargeGraphs
>>>
>>> I also note you are running a 3214 build which is from June 2015 and
>>> would recommend updating to the latest 3217 build from April 2016 …
>>>
>>> Best Regards
>>> Hugh Williams
>>> Professional Services
>>> OpenLink Software, Inc. // http://www.openlinksw.com/
>>> Weblog -- http://www.openlinksw.com/blogs/
>>> LinkedIn -- http://www.linkedin.com/company/openlink-software/
>>> Twitter -- http://twitter.com/OpenLink
>>> Google+ -- http://plus.google.com/100570109519069333827/
>>> Facebook -- http://www.facebook.com/OpenLinkSoftware
>>> Universal Data Access, Integration, and Management Technology Providers
>>>
>>> > On 2 Jul 2016, at 15:06, Pantelis Natsiavas <natsia...@gmail.com>
>>> wrote:
>>> >
>>> > Hi everybody.
>>> >
>>> > I am trying to execute the following query
>>> >
>>> > INSERT INTO GRAPH <targetGraph> {
>>> > ?a <someProperty> ?b
>>> > }
>>> > WHERE {
>>> > GRAPH <sourceGraph> {
>>> > ?a <someOtherProperty> ?c .
>>> > BIND(REPLACE(STR(?a),"something:", "", "i") AS ?b) }
>>> > }
>>> >
>>> > The sourceGraph contains about 94 million triplets. When I execute the
>>> above SPARQL, I get no error, however, I get no response either. I check
>>> after some minutes and the <targetGraph> has not yet been created. The
>>> system acts as if I had never executed the statement.
>>> >
>>> > I have tried executing the above through /sparql and through the
>>> /conductor web interfaces.
>>> >
>>> > Is there a way to see what is really going on? Am I doing something
>>> wrong?
>>> >
>>> > I have also already tried to debug through the isql-v localhost:8890
>>> dba <password> -D but I get a totally uncomprehensible error:
>>> >
>>> > isql-v() [0x44986a]
>>> > isql-v() [0x4498c8]
>>> > isql-v() [0x44b00e]
>>> > isql-v() [0x42fc65]
>>> > isql-v() [0x4108b0]
>>> > isql-v() [0x4043ee]
>>> > /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)
>>> [0x7fc06a645ec5]
>>> > isql-v() [0x40441e]
>>> > GPF: Dkernel.c:3080 can't free if in served sessions
>>> > Segmentation fault (core dumped)
>>> >
>>> > Please note that I am using Version: 07.20.3214, Build: Oct 14 2015 on
>>> ubuntu server.
>>> >
>>> > Kind regards,
>>> > Pantelis Natsiavas
>>> >
>>> ------------------------------------------------------------------------------
>>> > Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in San
>>> > Francisco, CA to explore cutting-edge tech and listen to tech
>>> luminaries
>>> > present their vision of the future. This family event has something for
>>> > everyone, including kids. Get more information and register today.
>>> >
>>> http://sdm.link/attshape_______________________________________________
>>> > Virtuoso-users mailing list
>>> > Virtuoso-users@lists.sourceforge.net
>>> > https://lists.sourceforge.net/lists/listinfo/virtuoso-users
>>>
>>>
>>
>> ------------------------------------------------------------------------------
>> Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in San
>> Francisco, CA to explore cutting-edge tech and listen to tech luminaries
>> present their vision of the future. This family event has something for
>> everyone, including kids. Get more information and register today.
>> http://sdm.link/attshape_______________________________________________
>> Virtuoso-users mailing list
>> Virtuoso-users@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/virtuoso-users
>>
>>
>>
> <virtuoso.log.part>
>
>
>
------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity planning
reports.http://sdm.link/zohodev2dev
_______________________________________________
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users
