FYI Paolo Castagna wrote: > This time, UNIX sort filled /tmp... > I'll try specifying the --temporary-directory=DIR or, better, via $TMPDIR > env variable (this way there is no need to change tdbloader2 script).
This time, I was able to load the Freebase data dump (converted into RDF) using tdbloader2 (which is included in TDB). This is how I run tdbloader2 using an EC2 m1.xlarge instance (i.e. 15 GB memory): export JVM_ARGS="-Xmx4096m -server" export TMPDIR=/mnt/data/tmp tdbloader2 --loc /mnt/data/freebase /mnt/data/freebase2rdf/freebase-datadump-rdf.nt.gz Total elapsed time to load 618,465,279 triples: ~12 hours (i.e. ~10,000 triples/s overall speed) This is the log: Mar 7 13:11:37 ip-10-54-167-166 build: 13:11:37 -- TDB Bulk Loader Start Mar 7 13:11:37 ip-10-54-167-166 build: 13:11:37 Data phase Mar 7 13:11:39 ip-10-54-167-166 build: Load: /mnt/data/freebase2rdf/freebase-datadump-rdf.nt.gz -- 2012/03/07 13:11:38 UTC Mar 7 13:11:42 ip-10-54-167-166 build: Add: 50,000 Data (Batch: 16,550 / Avg: 16,550) Mar 7 13:11:43 ip-10-54-167-166 build: Add: 100,000 Data (Batch: 39,184 / Avg: 23,272) [...] Mar 7 19:13:51 ip-10-54-167-166 build: Add: 618,450,000 Data (Batch: 53,078 / Avg: 28,457) Mar 7 19:17:01 ip-10-54-167-166 CRON[7725]: pam_unix(cron:session): session opened for user root by (uid=0) Mar 7 19:17:01 ip-10-54-167-166 CRON[7727]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Mar 7 19:17:01 ip-10-54-167-166 CRON[7725]: pam_unix(cron:session): session closed for user root Mar 7 19:24:44 ip-10-54-167-166 build: Total: 618,465,279 tuples : 22,385.15 seconds : 27,628.37 tuples/sec [2012/03/07 19:24:44 UTC] Mar 7 19:24:45 ip-10-54-167-166 build: 19:24:44 Index phase Mar 7 19:24:45 ip-10-54-167-166 build: 19:24:45 Index SPO Mar 7 21:03:18 ip-10-54-167-166 build: 21:03:18 Build SPO Mar 7 21:14:24 ip-10-54-167-166 build: 21:14:24 Index POS Mar 7 23:38:28 ip-10-54-167-166 build: 23:38:28 Build POS Mar 7 23:49:03 ip-10-54-167-166 build: 23:49:03 Index OSP Mar 8 00:56:13 ip-10-54-167-166 build: 00:56:13 Build OSP Mar 8 01:08:17 ip-10-54-167-166 build: 01:08:17 Index phase end Mar 8 01:08:59 ip-10-54-167-166 build: 01:08:59 -- TDB Bulk Loader Finish Mar 8 01:08:59 ip-10-54-167-166 build: 01:08:59 -- 43000 seconds Paolo
