I've made a dataset with about 10M nquads, 5-6 graphs, stored as a single .nq file. I've launched tdbloader2 to create a new dataset from this file, but I see a constant and remarkable slow down as more nquads are added to the dataset. Here are some INFO during processing:
INFO Add: 50,000 Data (Batch: 12,983 / Avg: 12,983) INFO Add: 500,000 Data (Batch: 77,639 / Avg: 51,743) INFO Add: 1,000,000 Data (Batch: 81,833 / Avg: 64,926) INFO Add: 2,000,000 Data (Batch: 84,745 / Avg: 72,745) INFO Add: 3,000,000 Data (Batch: 79,365 / Avg: 76,591) INFO Add: 4,000,000 Data (Batch: 91,575 / Avg: 77,605) INFO Add: 5,000,000 Data (Batch: 3,582 / Avg: 49,010) INFO Add: 6,000,000 Data (Batch: 3,915 / Avg: 22,031) INFO Add: 7,000,000 Data (Batch: 11,887 / Avg: 16,724) INFO Add: 8,000,000 Data (Batch: 4,121 / Avg: 15,455) INFO Add: 9,000,000 Data (Batch: 24,038 / Avg: 14,804) I wonder if this is normal or if there's anything I can do to speed this up.
