FYI

Paolo Castagna wrote:
> This time, UNIX sort filled /tmp...
> I'll try specifying the --temporary-directory=DIR or, better, via $TMPDIR
> env variable (this way there is no need to change tdbloader2 script).

This time, I was able to load the Freebase data dump (converted into RDF) using 
tdbloader2 (which is included in TDB).

This is how I run tdbloader2 using an EC2 m1.xlarge instance (i.e. 15 GB 
memory):

export JVM_ARGS="-Xmx4096m -server"
export TMPDIR=/mnt/data/tmp
tdbloader2 --loc /mnt/data/freebase 
/mnt/data/freebase2rdf/freebase-datadump-rdf.nt.gz

Total elapsed time to load 618,465,279 triples:
~12 hours (i.e. ~10,000 triples/s overall speed)

This is the log:
Mar  7 13:11:37 ip-10-54-167-166 build:  13:11:37 -- TDB Bulk Loader Start
Mar  7 13:11:37 ip-10-54-167-166 build:  13:11:37 Data phase
Mar  7 13:11:39 ip-10-54-167-166 build: Load: 
/mnt/data/freebase2rdf/freebase-datadump-rdf.nt.gz -- 2012/03/07 13:11:38 UTC
Mar  7 13:11:42 ip-10-54-167-166 build: Add: 50,000 Data (Batch: 16,550 / Avg: 
16,550)
Mar  7 13:11:43 ip-10-54-167-166 build: Add: 100,000 Data (Batch: 39,184 / Avg: 
23,272)
[...]
Mar  7 19:13:51 ip-10-54-167-166 build: Add: 618,450,000 Data (Batch: 53,078 / 
Avg: 28,457)
Mar  7 19:17:01 ip-10-54-167-166 CRON[7725]: pam_unix(cron:session): session 
opened for user root by (uid=0)
Mar  7 19:17:01 ip-10-54-167-166 CRON[7727]: (root) CMD (   cd / && run-parts 
--report /etc/cron.hourly)
Mar  7 19:17:01 ip-10-54-167-166 CRON[7725]: pam_unix(cron:session): session 
closed for user root
Mar  7 19:24:44 ip-10-54-167-166 build: Total: 618,465,279 tuples : 22,385.15 
seconds : 27,628.37 tuples/sec [2012/03/07 19:24:44 UTC]
Mar  7 19:24:45 ip-10-54-167-166 build:  19:24:44 Index phase
Mar  7 19:24:45 ip-10-54-167-166 build:  19:24:45 Index SPO
Mar  7 21:03:18 ip-10-54-167-166 build:  21:03:18 Build SPO
Mar  7 21:14:24 ip-10-54-167-166 build:  21:14:24 Index POS
Mar  7 23:38:28 ip-10-54-167-166 build:  23:38:28 Build POS
Mar  7 23:49:03 ip-10-54-167-166 build:  23:49:03 Index OSP
Mar  8 00:56:13 ip-10-54-167-166 build:  00:56:13 Build OSP
Mar  8 01:08:17 ip-10-54-167-166 build:  01:08:17 Index phase end
Mar  8 01:08:59 ip-10-54-167-166 build:  01:08:59 -- TDB Bulk Loader Finish
Mar  8 01:08:59 ip-10-54-167-166 build:  01:08:59 -- 43000 seconds

Paolo

Reply via email to