On 14/03/2023 08:56, Steven Blanchard wrote:
Hello,Yes, I do a one time operation on a non-existant folder. It's faster to do this than split input file and do multiple upload ?Yes, the loading is to a disk. Why does the performance of indexing decrease over time?
As the indexes grow in size, the efficiency of the OS file system cache drops.
In loader=phased, one index is created as the loaded, then the phases create the other indexes by copying the first index to the others ... which is in effect a sort, and when the OS file system cache becomes less effective, there is more disk I/O.
The xloader are designed to avoid (reduce) this. It uses an external sort program (Linux sort(1) - batching sorting) to get the information to load sequentially.
Andy
Thanks, StevenLe lun., mars 13 2023 at 21:26:55 +0000, Andy Seaborne <[email protected]> a écrit :On 13/03/2023 18:35, Simon Bin wrote:if you do a 1 time operation of non-existant => tdb, try xloader.xloader is good when loading to disk, rather than on to an SSD. Steven - is that what you are using? AndyOn Mon, 2023-03-13 at 16:08 +0100, Steven Blanchard wrote:Hello, I am currently loading data in ttl format with the command line tdb2.tdbloader (option --loader=phased ). This data consists of 1.25 billion tuples. The first step of loading is finish and the indexing begin. Since then, the indexing speed has been steadily decreasing until reaching an average speed of 651 while only 474 million triples have been indexed. With this speed, the indexing take several month. Do you have tips and solution to speed up the indexing speed? Could this be due to the input files? Thanks, Steven
