dr0ptp4kt added a comment.
Following roughly the procedure in P54284 <https://phabricator.wikimedia.org/P54284> to rename the Spark-produced graph files (and updating `loadData.sh` with `FORMAT=part-%05d-46f26ac6-0b21-4832-be79-d7c8709f33fb-c000.ttl.gz` and still having a `date` call after each `curl` in it), I kicked off an import of the scholarly article entity graph like so to see how it goes with a buffer capacity of 100000: ubuntu22:~/rdf/dist/target/service-0.3.138-SNAPSHOT$ date; time ./loadData.sh -n wdq -d /mnt/firehose/split_0/nt_wd_schol -s 0 -e 0 2>&1 | tee loadData.log; time ./loadData.sh -n wdq -d /mnt/firehose/split_0/nt_wd_schol 2>&1 | tee -a loadData.log Wed Apr 3 09:32:54 PM CDT 2024 Processing part-00000-46f26ac6-0b21-4832-be79-d7c8709f33fb-c000.ttl.gz <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd";><html><head><meta http-equiv="Content-Type" content="text/html;charset=UTF-8"><title>blazegraph™ by SYSTAP</title ></head ><body<p>totalElapsed=55629ms, elapsed=55584ms, connFlush=0ms, batchResolve=0, whereClause=0ms, deleteClause=0ms, insertClause=0ms</p ><hr><p>COMMIT: totalElapsed=61598ms, commitTime=1712198035155, mutationCount=7349689</p ></html >Wed Apr 3 09:33:56 PM CDT 2024 real 1m1.702s user 0m0.004s sys 0m0.006s Processing part-00001-46f26ac6-0b21-4832-be79-d7c8709f33fb-c000.ttl.gz <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd";><html><head><meta http-equiv="Content-Type" content="text/html;charset=UTF-8"><title>blazegraph™ by SYSTAP</title ></head ><body<p>totalElapsed=61251ms, elapsed=61251ms, connFlush=0ms, batchResolve=0, whereClause=0ms, deleteClause=0ms, insertClause=0ms</p ><hr><p>COMMIT: totalElapsed=71925ms, commitTime=1712198106800, mutationCount=7774048</p ></html >Wed Apr 3 09:35:08 PM CDT 2024 Processing part-00002-46f26ac6-0b21-4832-be79-d7c8709f33fb-c000.ttl.gz This is with the following values in `RWStore.properties` com.bigdata.btree.writeRetentionQueue.capacity=4000 com.bigdata.rdf.sail.bufferCapacity=100000 and the following variable in `loadData.sh` HEAP_SIZE=${HEAP_SIZE:-"31g"} TASK DETAIL https://phabricator.wikimedia.org/T359062 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dr0ptp4kt Cc: ssingh, bking, dr0ptp4kt, Aklapper, Danny_Benjafield_WMDE, S8321414, Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, EBjune, KimKelting, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
_______________________________________________ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
