dr0ptp4kt added a comment.
The run with with buffer at 100000**0** and heap size at 31g and queue
capacity at 4000 on the gaming-class desktop completed.
Processing wikidump-000001332.ttl.gz
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd";><html><head><meta
http-equiv="Content-Type"
content="text/html;charset=UTF-8"><title>blazegraph™ by SYSTAP</title
></head
><body<p>totalElapsed=13580ms, elapsed=13580ms, connFlush=0ms,
batchResolve=0, whereClause=0ms, deleteClause=0ms, insertClause=0ms</p
><hr><p>COMMIT: totalElapsed=266483ms, commitTime=1711304860167,
mutationCount=4772590</p
></html
>Sun Mar 24 01:27:45 PM CDT 2024
real 5690m30.371s
... which is 3.95 days. I'm trying again, but going back to the buffer
capacity at 100000 instead of 100000**0** for one last comparison with these
runs on this subset of munged data, and without any larger pause between
batches of files (remember the previous run with buffer capacity at 100000 and
31g heap and queue capacity at 4000 was done by first running files 1-150, then
after coming back to the terminal sometime later resumed from file 151; but in
the real world we usually hope to just let this thing run one file after
another without any pause...in practice it could be that allowing the JVM time
to heal itself created some artificial speed gains, but we'll see).
Starting on Friday, March 29, 2024 at 1:40 PM CT...
ubuntu22:~/rdf/dist/target/service-0.3.138-SNAPSHOT$ time ./loadData.sh -n
wdq -d /mnt/firehose/munge_on_later_data_set -s 1 -e 1332
Processing wikidump-000000001.ttl.gz
I'll update when it's done. It should complete presumably sometime in the
next 24 hours.
TASK DETAIL
https://phabricator.wikimedia.org/T359062
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: dr0ptp4kt
Cc: ssingh, bking, dr0ptp4kt, Aklapper, Danny_Benjafield_WMDE, S8321414,
Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz,
Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, EBjune, KimKelting,
LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
_______________________________________________
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
