On 2 Sep 2014, at 00:01, Hugh Williams <hwilli...@openlinksw.com> wrote:
>>> Development indicate your suggestion is not without merit but >>> implementation is not as simple as it may seems as the indexes are not all >>> sequential, but something like that could possibly be implemented. It is >>> suggested you could try dropping the indexes on RDF_QUAD table, load the >>> Freebase datasets and then recreate indexes after loading, which would >>> require a smaller working set that would better fix into the 32GB RAM >>> available. The command for dropping the necessary indexes are: >>> >>> drop index rdf_quad_pogs; >>> drop index rdf_quad_sp; >>> drop index rdf_quad_op; >>> drop index rdf_quad_gs; >>> >>> and the respective indexes can then be recreated as detailed at: >>> >>> >>> http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtRDFPerformanceTuning?#RDF%20Index%20Scheme >>> >>> Note you need to recreate the column-wise indexes being v7. Let us know how >>> this works for you. >> >> Cool, will try. > > [Hugh] OK, let us know the outcome ... After 5 days: [Sun Sep 7 23:43:31 2014] virtuoso-t[11495]: segfault at ffffffffffffffff ip 00000000008c3a8e sp 00007f4ec2f79d20 error 7 in virtuoso-t[400000+b47000] yay... Performance also breaks down at some point with dropped indexes and 2 run_rdf_loaders as suggested. I'll append the output of `select * from DB.DBA.LOAD_LIST;` but for now i give up... Cheers, Jörn ll_file ll_graph ll_state ll_started ll_done ll_host ll_work_time ll_error VARCHAR NOT NULL VARCHAR INTEGER TIMESTAMP TIMESTAMP INTEGER INTEGER VARCHAR _______________________________________________________________________________ /usr/local/data/datasets/remote/freebase/2014-08-20/splitted/freebase-rdf-2014-08-17-00-00.aa.nt.gz http://rdf.freebase.com 2 2014.9.2 17:20.13 927132000 2014.9.2 17:34.28 11121000 0 NULL NULL /usr/local/data/datasets/remote/freebase/2014-08-20/splitted/freebase-rdf-2014-08-17-00-00.ab.nt.gz http://rdf.freebase.com 2 2014.9.2 17:20.59 818941000 2014.9.2 17:35.42 994563000 0 NULL NULL /usr/local/data/datasets/remote/freebase/2014-08-20/splitted/freebase-rdf-2014-08-17-00-00.ac.nt.gz http://rdf.freebase.com 2 2014.9.2 17:34.28 46244000 2014.9.2 17:52.1 42050000 0 NULL NULL /usr/local/data/datasets/remote/freebase/2014-08-20/splitted/freebase-rdf-2014-08-17-00-00.ad.nt.gz http://rdf.freebase.com 2 2014.9.2 17:35.43 10491000 2014.9.2 17:53.58 266217000 0 NULL NULL /usr/local/data/datasets/remote/freebase/2014-08-20/splitted/freebase-rdf-2014-08-17-00-00.ae.nt.gz http://rdf.freebase.com 2 2014.9.2 17:52.1 45510000 2014.9.2 18:11.45 21522000 0 NULL NULL /usr/local/data/datasets/remote/freebase/2014-08-20/splitted/freebase-rdf-2014-08-17-00-00.af.nt.gz http://rdf.freebase.com 2 2014.9.2 17:53.58 270750000 2014.9.2 18:14.13 266480000 0 NULL NULL /usr/local/data/datasets/remote/freebase/2014-08-20/splitted/freebase-rdf-2014-08-17-00-00.ag.nt.gz http://rdf.freebase.com 2 2014.9.2 18:11.45 25765000 2014.9.2 18:29.32 312824000 0 NULL NULL /usr/local/data/datasets/remote/freebase/2014-08-20/splitted/freebase-rdf-2014-08-17-00-00.ah.nt.gz http://rdf.freebase.com 2 2014.9.2 18:14.13 271520000 2014.9.2 18:34.51 216078000 0 NULL NULL /usr/local/data/datasets/remote/freebase/2014-08-20/splitted/freebase-rdf-2014-08-17-00-00.ai.nt.gz http://rdf.freebase.com 2 2014.9.2 18:29.32 321036000 2014.9.2 18:54.37 54526000 0 NULL NULL /usr/local/data/datasets/remote/freebase/2014-08-20/splitted/freebase-rdf-2014-08-17-00-00.aj.nt.gz http://rdf.freebase.com 2 2014.9.2 18:34.51 220487000 2014.9.2 18:54.44 130952000 0 NULL NULL /usr/local/data/datasets/remote/freebase/2014-08-20/splitted/freebase-rdf-2014-08-17-00-00.ak.nt.gz http://rdf.freebase.com 2 2014.9.2 18:54.37 59593000 2014.9.2 19:25.41 495402000 0 NULL NULL /usr/local/data/datasets/remote/freebase/2014-08-20/splitted/freebase-rdf-2014-08-17-00-00.al.nt.gz http://rdf.freebase.com 2 2014.9.2 18:54.44 143345000 2014.9.2 19:25.43 474136000 0 NULL NULL /usr/local/data/datasets/remote/freebase/2014-08-20/splitted/freebase-rdf-2014-08-17-00-00.am.nt.gz http://rdf.freebase.com 2 2014.9.2 19:25.41 660564000 2014.9.2 20:17.57 688038000 0 NULL NULL /usr/local/data/datasets/remote/freebase/2014-08-20/splitted/freebase-rdf-2014-08-17-00-00.an.nt.gz http://rdf.freebase.com 2 2014.9.2 19:25.43 475097000 2014.9.2 20:17.45 149151000 0 NULL NULL /usr/local/data/datasets/remote/freebase/2014-08-20/splitted/freebase-rdf-2014-08-17-00-00.ao.nt.gz http://rdf.freebase.com 2 2014.9.2 20:17.45 251450000 2014.9.3 1:1.5 419037000 0 NULL NULL /usr/local/data/datasets/remote/freebase/2014-08-20/splitted/freebase-rdf-2014-08-17-00-00.ap.nt.gz http://rdf.freebase.com 2 2014.9.2 20:17.57 691065000 2014.9.3 0:44.23 531234000 0 NULL NULL /usr/local/data/datasets/remote/freebase/2014-08-20/splitted/freebase-rdf-2014-08-17-00-00.aq.nt.gz http://rdf.freebase.com 2 2014.9.3 0:44.23 819381000 2014.9.3 7:35.28 175575000 0 NULL NULL /usr/local/data/datasets/remote/freebase/2014-08-20/splitted/freebase-rdf-2014-08-17-00-00.ar.nt.gz http://rdf.freebase.com 2 2014.9.3 1:1.5 556141000 2014.9.3 7:42.26 712562000 0 NULL NULL /usr/local/data/datasets/remote/freebase/2014-08-20/splitted/freebase-rdf-2014-08-17-00-00.as.nt.gz http://rdf.freebase.com 2 2014.9.3 7:35.28 443083000 2014.9.4 8:41.4 23807000 0 NULL NULL /usr/local/data/datasets/remote/freebase/2014-08-20/splitted/freebase-rdf-2014-08-17-00-00.at.nt.gz http://rdf.freebase.com 2 2014.9.3 7:42.26 781877000 2014.9.4 8:34.42 373954000 0 NULL NULL /usr/local/data/datasets/remote/freebase/2014-08-20/splitted/freebase-rdf-2014-08-17-00-00.au.nt.gz http://rdf.freebase.com 2 2014.9.4 8:34.42 519416000 2014.9.5 19:47.29 159439000 0 NULL NULL /usr/local/data/datasets/remote/freebase/2014-08-20/splitted/freebase-rdf-2014-08-17-00-00.av.nt.gz http://rdf.freebase.com 2 2014.9.4 8:41.4 30914000 2014.9.7 10:45.38 995297000 0 NULL NULL /usr/local/data/datasets/remote/freebase/2014-08-20/splitted/freebase-rdf-2014-08-17-00-00.aw.nt.gz http://rdf.freebase.com 1 2014.9.5 19:47.29 347568000 NULL 0 NULL NULL /usr/local/data/datasets/remote/freebase/2014-08-20/splitted/freebase-rdf-2014-08-17-00-00.ax.nt.gz http://rdf.freebase.com 1 2014.9.7 10:45.39 358793000 NULL 0 NULL NULL /usr/local/data/datasets/remote/freebase/2014-08-20/splitted/freebase-rdf-2014-08-17-00-00.ay.nt.gz http://rdf.freebase.com 0 NULL NULL NULL NULL NULL /usr/local/data/datasets/remote/freebase/2014-08-20/splitted/freebase-rdf-2014-08-17-00-00.az.nt.gz http://rdf.freebase.com 0 NULL NULL NULL NULL NULL /usr/local/data/datasets/remote/freebase/2014-08-20/splitted/freebase-rdf-2014-08-17-00-00.ba.nt.gz http://rdf.freebase.com 0 NULL NULL NULL NULL NULL 27 Rows. -- 27 msec. ------------------------------------------------------------------------------ Want excitement? Manually upgrade your production database. When you want reliability, choose Perforce Perforce version control. Predictably reliable. http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk _______________________________________________ Virtuoso-users mailing list Virtuoso-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/virtuoso-users
