[
https://issues.apache.org/jira/browse/JENA-1552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16499165#comment-16499165
]
ASF GitHub Bot commented on JENA-1552:
--------------------------------------
GitHub user afs opened a pull request:
https://github.com/apache/jena/pull/426
JENA-1552: Phased loader
This has become a general purpose loader (`LoaderMain`) which is controlled
by a `LoaderPlan`. LoaderMain is both the parallel loader and the phased loader
of JENA-1552.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/afs/jena phased-loader
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/jena/pull/426.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #426
----
commit 3b712638db710630882844fd4c266120fa2a49ca
Author: Andy Seaborne <andy@...>
Date: 2018-06-01T13:07:55Z
JENA-1552: Phased loader
----
> Bulk loader for TDB2 (phased loading)
> -------------------------------------
>
> Key: JENA-1552
> URL: https://issues.apache.org/jira/browse/JENA-1552
> Project: Apache Jena
> Issue Type: Improvement
> Components: TDB2
> Reporter: Andy Seaborne
> Assignee: Andy Seaborne
> Priority: Major
>
> Following on from JENA-1550, this ticket is for phased loading which combined
> features of the sequential loader and the parallel loader.
> When building all the persistent datastructures (parallel loader), the work
> on different indexes at the same time is competing for hardware resources,
> RAM and I/O bandwidth. As the size to load grows, this becomes a noticeable
> slowdown.
> The sequential loader is the other extreme of the design spectrum. It does
> work on one index at a time so as to maximize caching efficiency.
> Phased loading has parallel operation per phase and splits work into subsets
> of indexes.
> At 200m and loading to rotational disk, an experimental phased loader working
> with 2 indexes at a time, starts to become faster than parallel on the same
> hardware as used for the [figures in
> JENA-1550|https://issues.apache.org/jira/browse/JENA-1550#comment-16484269]
> (57K parallel, 70K phased).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
