dr0ptp4kt added a comment.
**More about NVMe versus SSD**
Runs were also done to see the effects on 150 munged files (out of a set of
2202 files) from the full Wikidata import, which allows for exercising more
disk related pieces. This was tried with both types of target disk - SATA SSD
and M.2 NVMe - on the 2018 gaming desktop. This was done with the
`bufferCapacity` of 100000.
The M.2 NVMe was faster, somewhere between 16%-19% faster.
Notice in the following commands the paths
- `~/rdf`, which is part of a mount on the NVMe
- `/mnt/t`, which is a copy of `~/rdf`, but on a SATA SSD
- `/mnt/firehose/`, yet another SATA SSD, bearing the full set of munged files
**Target is NVMe**
ubuntu22:~/rdf/dist/target/service-0.3.138-SNAPSHOT$ time ./loadData.sh -n
wdq -d /mnt/firehose/munge_on_later_data_set -s 1 -e 150
...
>Processing wikidump-000000150.ttl.gz
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd";><html><head><meta
http-equiv="Content-Type"
content="text/html;charset=UTF-8"><title>blazegraph™ by SYSTAP</title
></head
><body<p>totalElapsed=33999ms, elapsed=33999ms, connFlush=0ms,
batchResolve=0, whereClause=0ms, deleteClause=0ms, insertClause=0ms</p
><hr><p>COMMIT: totalElapsed=76005ms, commitTime=1709099819611,
mutationCount=3098484</p
></html
>
real 319m50.828s
**Target is SATA SSD, run attempt 1**
Now, the SATA SSD as the target (as before, the source has been a different
SATA SSD).
ubuntu22:/mnt/t/rdf/dist/target/service-0.3.138-SNAPSHOT$ time
./loadData.sh -n wdq -d /mnt/firehose/munge_on_later_data_set -s 1 -e 150
>Processing wikidump-000000150.ttl.gz
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd";><html><head><meta
http-equiv="Content-Type"
content="text/html;charset=UTF-8"><title>blazegraph™ by SYSTAP</title
></head
><body<p>totalElapsed=45665ms, elapsed=45665ms, connFlush=0ms,
batchResolve=0, whereClause=0ms, deleteClause=0ms, insertClause=0ms</p
><hr><p>COMMIT: totalElapsed=114606ms, commitTime=1709141576293,
mutationCount=3098484</p
></html
>
real 381m19.703s
So, the SATA SSD as target yielded a result about 19% slower.
**Target is SATA SSD, run attempt 2**
The SATA SSD target was tried this again from the same directory (as always,
first stopping Blazegraph and deleting the journal) again just to get a feeling
of whether this wasn't a fluke on the SATA SSD.
ubuntu22:/mnt/t/rdf/dist/target/service-0.3.138-SNAPSHOT$ time
./loadData.sh -n wdq -d /mnt/firehose/munge_on_later_data_set -s 1 -e 150
><body<p>totalElapsed=46490ms, elapsed=46490ms, connFlush=0ms,
batchResolve=0, whereClause=0ms, deleteClause=0ms, insertClause=0ms</p
><hr><p>COMMIT: totalElapsed=120472ms, commitTime=1709169683880,
mutationCount=3098484</p
></html
>
real 373m52.079s
Still, some 16.5% slower on the SSD.
TASK DETAIL
https://phabricator.wikimedia.org/T359062
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: dr0ptp4kt
Cc: ssingh, bking, dr0ptp4kt, Aklapper, Danny_Benjafield_WMDE, Astuthiodit_1,
karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, Nandana, Lahi,
Gq86, GoranSMilovanovic, QZanden, EBjune, KimKelting, LawExplorer, _jensen,
rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
_______________________________________________
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
