Hi Hugh,
On Tue, Oct 27, 2009 at 15:33, Hugh Williams <[email protected]>wrote:
> Can you please provide a copy of your Virtuoso Configuration file
> (virtuoso.ini) and Virtuoso log file (virtuoso.log) for review.
>
The log shows nothing unusual - the usual startup/roll forward and then
nothing but checkpoints. I've included the ini at the end if this email.
I've not had tracing turned on before the point the problems occur yet.
Which trace_on arguments are most helpful in debugging? I could perhaps
leave them enabled in the ini file for a while.
Also, can you please clarifiy if:
>
1. This problem has only started occurring since your upgrade to the new
> 6.0.0 release as it seems unclear from your mails ?
>
I've only noticed the problem since upgrading but we hadn't done anything
significant other than import a selection of the rdf files beforehand.
Therefore I can't say with any certainty that it worked properly in TP1.
2. Do the hangs you encounter only occur when accessing the server via ODBC
> ? Also what ODBC operations are typically being performed against the
> Virtuoso Server ?
>
Again, the hangs were only noticed when previously working pyodbc sparql
queries were timing out, but I can't be certain that when left to its own
devices virtuoso wasn't behaving strangely before.
> Should anymore deadlocks start occurring can you provide the output of
> running the following query from the isql prompt:
>
> select top 10 * from sys_l_stat order by deadlocks desc;
>
Will do.
I managed to crash virtuoso with a segfault by importing the 25GB /
228625909 line uniprot.uniprot.n3 from bio2rdf. I then compiled virtuoso
with debugging enabled and starting with a clean database, imported
uniprot.uniprot.n3 again using the following commands:
OpenLink Interactive SQL (Virtuoso), version 0.9849b.
Type HELP; for help and EXIT; to exit.
SQL>
checkpoint_interval(-1);
Connected to OpenLink Virtuoso
Driver: 06.00.3123 OpenLink Virtuoso ODBC Driver
Done. -- 0 msec.
SQL> DB.DBA.TTLP_MT (file_to_string_output
('/opt/bio2rdf/data/uniprot.uniprot.n3'), '', '
http://bio2rdf.org/uniprot.uniprot#', 128);
*** Error 08S01: [Virtuoso Driver]CL065: Lost connection to server
at line 2 of Top-Level:
DB.DBA.TTLP_MT (file_to_string_output
('/opt/bio2rdf/data/READY/uniprot.uniprot.n3'), '', '
http://bio2rdf.org/uniprot.uniprot#', 128)
The backtrace from virtuoso is:
16:58:21 GPF: row.c:2319 row fill overflow max bytes
> GPF: row.c:2319 row fill overflow max bytes
>
> Program received signal SIGSEGV, Segmentation fault.
> [Switching to Thread 0x7fff2dec9910 (LWP 24281)]
> gpf_notice (file=0x807683 "row.c", line=2319, text=0x806221 "row fill
> overflow max bytes") at Dkutil.c:62
> 62 *(long *) -1 = -1;
>
> ----
>
> (gdb) bt
> #0 gpf_notice (file=0x807683 "row.c", line=2319, text=0x806221 "row fill
> overflow max bytes") at Dkutil.c:62
> #1 0x00000000004b3d47 in row_set_col (rf=0x7fff2debb000, cl=0x2a51770,
> data=<value optimized out>) at row.c:2319
> #2 0x00000000004a7bf2 in page_set_values (buf=<value optimized out>,
> rf=<value optimized out>, key=0x15904b0, irow=<value optimized out>,
> values=<value optimized out>, leaf=<value optimized out>) at page.c:807
> #3 0x00000000004a83b5 in pf_rd_append (pf=0x7fff2debc0e0,
> rd=0x7fff2dec3210, split_after=0x7fff2debc12e) at page.c:1834
> #4 0x00000000004a8e7b in page_apply_1 (itc=<value optimized out>,
> buf=0x7ffff62123f8, n_delta=1, delta=<value optimized out>, op=<value
> optimized out>, paf=0x7fff2debc280) at page.c:2324
> #5 0x00000000004a8fba in page_apply_1 (itc=<value optimized out>,
> buf=0x7ffff62123f8, n_delta=1, delta=<value optimized out>, op=<value
> optimized out>, paf=0x7fff2debc280) at page.c:2227
> #6 0x00000000004a9bd2 in page_apply_s (itc=0x7ffff6f58860, buf=0x0,
> n_delta=0, delta=0x2, op=770480400) at page.c:2383
> #7 0x00000000004a9ce5 in page_apply (itc=0x7fff2dec7ae0,
> buf=0x7ffff62123f8, n_delta=1, delta=0x7fff2dec7918, op=0) at page.c:2400
> #8 0x000000000047dbb5 in itc_insert_dv (it=0x7ffff6f58860, buf_ret=<value
> optimized out>, rd=0x7fff2dec79f0, is_recursive=<value optimized out>,
> new_rl=<value optimized out>) at insert.c:603
> #9 0x000000000047e5e3 in itc_insert_unq_ck (it=0x7fff2dec7ae0,
> rd=0x7fff2dec79f0, unq_buf=0x7fff2dec7a60) at insert.c:665
> #10 0x00000000004ba3a7 in key_insert (ins=<value optimized out>,
> qst=0x70983e8, it=0x7fff2dec7ae0, ik=0x6947ff0) at row.c:2619
> #11 0x000000000055d757 in insert_node_run (ins=<value optimized out>,
> inst=<value optimized out>, state=<value optimized out>) at sqlrun.c:1621
> #12 0x000000000055d956 in insert_node_input (ins=0x694a250, inst=0x70983e8,
> state=0x70983e8) at sqlrun.c:1660
> #13 0x0000000000536171 in ins_qnode (ins=<value optimized out>, qst=<value
> optimized out>) at sqlintrp.c:1503
> #14 0x0000000000539d43 in code_vec_run_1 (code_vec=<value optimized out>,
> qst=<value optimized out>, offset=<value optimized out>) at sqlintrp.c:2008
> #15 0x0000000000557246 in qn_input (xx=0x7fff28d09fe0, inst=0x70983e8,
> state=0x70983e8) at sqlrun.c:680
> #16 0x000000000055b44f in qr_exec (cli=<value optimized out>, qr=<value
> optimized out>, caller=<value optimized out>, cr_name=0x0, stmt=0x6caf1e0,
> lc_ret=<value optimized out>, parms=0x7fff22795c08, opts=0x0,
> named_params=0) at sqlrun.c:3167
> #17 0x00000000006de512 in aq_sql_func (av=0x7fff2b392b08,
> err_ret=0x7fff2a5dab88) at aqueue.c:448
> #18 0x00000000006df6e4 in aq_thread_func (aqt=0x6a25b10) at aqueue.c:89
> #19 0x00000000007974df in _thread_boot (arg=0x7fff289b6dc0) at
> sched_pthread.c:290
> #20 0x00007ffff75f4a04 in start_thread () from /lib/libpthread.so.0
> #21 0x00007ffff6cce7bd in clone () from /lib/libc.so.6
> #22 0x0000000000000000 in ?? ()
>
> ----
>
> (gdb) info threads
> 24 Thread 0x7fff2de3f910 (LWP 24282) 0x00007ffff75f95a9 in
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
> * 23 Thread 0x7fff2dec9910 (LWP 24281) gpf_notice (file=0x807683 "row.c",
> line=2319, text=0x806221 "row fill overflow max bytes") at Dkutil.c:62
> 22 Thread 0x7fff2df53910 (LWP 24280) 0x00007ffff75f95a9 in
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
> 21 Thread 0x7fff2dfdd910 (LWP 24279) 0x00007ffff75f95a9 in
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
> 20 Thread 0x7fff2e067910 (LWP 24260) 0x00007ffff75f95a9 in
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
> 19 Thread 0x7fff2e0f1910 (LWP 24258) 0x00007ffff75f95a9 in
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
> 18 Thread 0x7fff2e17b910 (LWP 24257) 0x00007ffff75f95a9 in
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
> 17 Thread 0x7fff2e205910 (LWP 24256) 0x00007ffff75f95a9 in
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
> 16 Thread 0x7fff2e28f910 (LWP 24255) 0x00007ffff75f95a9 in
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
> 15 Thread 0x7fff2e319910 (LWP 24254) 0x00007ffff75f95a9 in
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
> 14 Thread 0x7fff2e3a3910 (LWP 24253) 0x00007ffff75f95a9 in
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
> 13 Thread 0x7fff2e42d910 (LWP 24252) 0x00007ffff75f95a9 in
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
> 12 Thread 0x7fff2e4b7910 (LWP 24251) 0x00007ffff75f95a9 in
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
> 11 Thread 0x7fff2e541910 (LWP 24250) 0x00007ffff75f95a9 in
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
> 10 Thread 0x7fff2e5cb910 (LWP 24249) 0x00007ffff75f95a9 in
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
> 9 Thread 0x7ffff7e93910 (LWP 24248) 0x00007ffff6cc73c2 in select () from
> /lib/libc.so.6
> 8 Thread 0x7ffff7ead910 (LWP 24247) 0x00007ffff75f95a9 in
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
> 7 Thread 0x7fff2e655910 (LWP 24246) 0x00007ffff75f95a9 in
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
> 6 Thread 0x7fff2e6df910 (LWP 24245) 0x00007ffff75f95a9 in
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
> 3 Thread 0x7ffff7f2b910 (LWP 24242) 0x00007ffff75f95a9 in
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
> 2 Thread 0x7fff2fdd0910 (LWP 24241) 0x00007ffff75f95a9 in
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
> 1 Thread 0x7ffff7ff06f0 (LWP 24240) 0x00007ffff75f95a9 in
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
>
The virtuoso.ini being used for both problems:
;
> ; virtuoso.ini
> ;
> ; Configuration file for the OpenLink Virtuoso VDBMS Server
> ;
> ;
> ; Database setup
> ;
> [Database]
> DatabaseFile = utopia-database.db
> TransactionFile = utopia-database.trx
> ErrorLogFile = utopia-database.log
> ErrorLogLevel = 7
> FileExtend = 200
> LogSegments = 0
> ;crashdump_start_dp = 0
> ;crashdump_end_dp = 0
> Striping = 0
> TempStorage = TempDatabase
> Syslog = 1
>
> ; See http://docs.openlinksw.com/virtuoso/dbadm.html
> [Parameters]
> ;TraceOn = soap, errors
> O_DIRECT = 0
> UseAIO = 1
> ; This specifies whether to use asynchronous file I/O on supporting
> ; Unix systems. A value of 0 means not using it. A value of 1 means
> ; using lio_listio for any background write or read ahead if
> ; available. A value of 2 means to use the regular blocking read and
> ; write but to merge adjacent operations into a single system call
> ; when possible.
> ServerPort = 1111
> ServerThreads = 100
> CheckpointInterval = 10
> AutoCheckpointLogSize = 1000000
> NumberOfBuffers = 400000
> MaxDirtyBuffers = 225000
> TransactionAfterImageLimit = 1500000000
> MaxCheckpointRemap = 300000
> UnremapQuota = 0
> AtomicDive = 1
> PrefixResultNames = 0
> CaseMode = 2
> DisableMtWrite = 0
> MaxStaticCursorRows = 5000
> CheckpointAuditTrail = 0
> AllowOSCalls = 0
> SchedulerInterval = 10
> FreeTextBatchSize = 50000000
> ;NullUnspecifiedParams = 1
> ;MinAutoCheckpointSize = 10000
> ;CheckpointAuditTrail = 0
> ;CheckpointSyncMode = 2
> ;ServerThreadSize = 500000
> ;MainThreadSize = 1000000
> ;FutureThreadSize = 1000000
> ;Collation = spanish
> ;Charset = UTF-8
> ;Charset = WINDOWS-1251
> DirsAllowed = ., /opt/virtuoso/vad,
> /opt/virtuoso/virtuoso-opensource-V6x/vad, /opt/utopia/data, ./backup,
> /opt/bio2rdf
> VADInstallDir = /opt/virtuoso/vad
> DbaExecutables = /bin/tar,/bin/sh
> ;SafeExecutables = /home/virtdev/distrib/db/dbpump,
> /home/virtdev/distrib/db/rm, ../bin/isql
> SafeExecutables = ./gpf.out
> ;SQLOptimizer = 1
> ;SSLServerPort = 1113
> SSLCertificate = ./keys/srv.cert.pem
> SSLPrivateKey = ./keys/srv.key.pem.ne
> ;X509ClientVerify = 1
> ;X509ClientVerifyCAFile = ./keys/ca.pem
> ;X509ClientVerifyDepth = 15
> ;X509ExtensionOID = 2.16.840.1.1113.1
> ;ThreadCleanupInterval = 1
> ;ThreadThreshold = 0
> CallstackOnException = 1
> PLDebug = 2
> ;TestCoverage = cov.xml
> ;TraceOn = errors
> MaxOptimizeLayouts = 1000
> TempDir = /var/tmp
> ;RunAsUser = nobody
> ;JavaClasspath =
> /home/imitko/bin/j2sdk1.4.0_01/lib/tools.jar:/home/imitko/download/jakarta-tomcat-4.1.18/bin/bootstrap.jar
> ;JavaVMOption1 =
> -Djava.endorsed.dirs=/home/imitko/download/jakarta-tomcat-4.1.18/bin:/home/imitko/download/jakarta-tomcat-4.1.18/common/endorsed
> ;JavaVMOption2 =
> -Dcatalina.base=/home/imitko/download/jakarta-tomcat-4.1.18
> ;JavaVMOption3 =
> -Dcatalina.home=/home/imitko/download/jakarta-tomcat-4.1.18
> ;JavaVMOption4 =
> -Djava.io.tmpdir=/home/imitko/download/jakarta-tomcat-4.1.18/temp
> ;JavaVMOption5 = -Ddeprecated.j2ee.class.path= -start
> ;JavaClasspath =
> /home/imitko/virtuoso/db:/home/imitko/virtuoso/binsrc/vsp/tutorial/services/so_s_30
> ;SkipStartupCompilation = 0
> SqlWarningMode = 1
> MinSignalHandling = 1
> ;DbevEnable = 0
>
> [HTTPServer]
> ServerPort = 8890
> ServerRoot = vsp
> ServerThreads = 10
> MaxKeepAlives = 10
> KeepAliveTimeout = 10
> MaxCachedProxyConnections = 10
> ProxyConnectionCacheTimeout = 10
> DavRoot = DAV
> ;SSLPort = 4443
> SSLCertificate = ./keys/srv/cert.pem
> SSLPrivateKey = ./keys/srv/keyn.pem
> X509ClientVerifyCAFile = ./keys/srv/ca.pem
> ;X509ClientVerify = 1
> ;X509ClientVerifyDepth = 15
> ;DefaultMailServer = gw.gmz:25
> ;HTTPProxyEnabled = 1
> HTTPLogFile = logs/http29102009.log
> EnabledGzipContent = 1
> Charset = utf-8
> ;POP3Port = 1100
> ;NewsServerPort = 1235
> ;ServerIdString = Apache/1.3.9 (Unix) (Red Hat/Linux) DAV/1.0.3
> PHP/3.0.16
> ;HTTPThreadSize = 10000000
> ;DAVQuotaEnabled = 1
> ;EnableRequestTrap = 1
> ;SpamFilter = 1
> ;FTPServerPort = 2211
> FTPServerLogFile = logs/ftp16062006.log
> ;ClientIdString = Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.8)
> Gecko/20050524
> ;DAVChunkedQuota = 1000000000
> HttpSessionSize = 20971520
> ;MaintenancePage = atomic.html
>
> [AutoRepair]
> BadParentLinks = 0
> BadDTP = 0
>
> [Client]
> SQL_QUERY_TIMEOUT = 0
> SQL_TXN_TIMEOUT = 0
> SQL_PREFETCH_ROWS = 100
> SQL_PREFETCH_BYTES = 16000
> ;SQL_NO_CHAR_C_ESCAPE = 1
> ;SQL_ENCRYPTION_ON_PASSWORD = 2
> ;SQL_ENCRYPTION_ON_PASSWORD = 1
>
> [VDB]
> ArrayOptimization = 1
> NumArrayParameters = 100
> ;VDBDisconnectTimeout = 1
> ;VDBOracleCatalogFix = 1
> ;AttachInAutoCommit = 1
> TrimTrailingSpacesForDSN = inf7ma
> ;DisableVDBStatisticsRefresh = 0
>
> ;
> ; Striping setup
> ;
> ; These parameters have only effect when StripiNg is set to 1 in the
> ; [Database] section, in which case the DatabaseFile parameter is ignored.
> ;
> ; With striping, the database is spawned accross multiple segments
> ; where each segment can have multiple stripes.
> ;
> ; Format of the lines below:
> ; Segment<number> = <size>, <stripe file name> [, <stripe file name> ..
> ]
> ;
> ; <number> must be ordered from 1 up.
> ;
> ; The <size> is the total size of the segment which is equally divided
> ; accross all stripes spawning the segment. Its specification can be in
> ; gigabytes (g), megabytes (m), kilobytes (k) or in database blocks
> ; (b, the default)
> ;
> ; Note that the segment size must be a multiple of the database page size
> ; which is currently 4k. Also, the segment size must be divisible by the
> ; number of stripe files spawning the segment.
> ;
> ; The example below creates a 200 meg database striped on two segments
> ; with two stripes of 50 meg and one of 100 meg.
> ;
> ; You can always add more segments to the configuration, but once
> ; added, do not change the setup.
> ;
> ;[Striping]
> ;Segment1 = 50M, utopia-temp-1.seg
> ;Segment2 = 50M, utopia-temp-2.seg
> ;Segment3 = 50M, utopia-temp-3.seg
> ;Log1 = utopia-temp-1.lgs 50M
> ;Log2 = utopia-temp-2.lgs 50M
> ;Log3 = utopia-temp-3.lgs 50M
> ;[Replication]
> ;ServerName = utopiadb
> ;ServerEnable = 1
> ;QueueMax = 50000
> [TempDatabase]
> DatabaseFile = utopia-database.tdb
> TransactionFile = utopia-database.ttr
> FileExtend = 200
> ;[Mono]
> ;virtclr.dll = /home/imitko/virtuoso/binsrc/tests/biftest/virtclr.dll
> ;MONO_ROOT = /home/virtuoso/mono/lib
> ;MONO_PATH =
> /home/virtuoso/binsrc/tests/biftest/tests:/home/virtuoso/binsrc/tutorial/hosting/ho_s_4:/home/virtuoso/binsrc/tutorial/hosting/ho_s_5
> ;MONO_CFG_DIR = /home/virtuoso/mono/lib
> ;MONO_TRACE = On
>
> [Zero Config]
> ServerName = Utopia Server
> ;ServerDSN = UID=dba;PWD=dba
> ;SSLServerName = Linux Virtuoso/SSL
>
> [!Plugins]
> LoadPath = /opt/virtuoso/virtuoso-opensource-V6x/hosting
> Load1 = plain, wikiv
> Load2 = plain, im
> Load3 = plain, wbxml2
> Load4 = Hosting, hosting_perl.so
> ;Load4 = Hosting, hosting_python.so
>
> [URIQA]
> DynamicLocal = 0
> DefaultHost = localhost:8890
> ;LocalHostNames =
> ;LocalHostMasks =
> ;Fingerprint =
>
> [SPARQL]
> ;ExternalQuerySource = 1
> ;ExternalXsltSource = 1
> DefaultGraph = http://bio2rdf.org#
> DefaultQuery = select distinct ?s ?Concept where {?s a
> ?Concept} limit 100
> MaxCacheExpiration = 1 ; Cache Expiration time in seconds that
> overrides Sponger's default cache invalidation scheme
> ExternalQuerySource = 1
> ExternalXsltSource = 1
> ResultSetMaxRows = 100000
> MaxQueryCostEstimationTime = 10000 ; in seconds
> MaxQueryExecutionTime = 30 ; in seconds
> DeferInferenceRulesInit = 1 ; Defer Loading of inference rules at
> start up
>
Best wishes,
James
