On Monday, 14 September 2015 at 13:10:50 UTC, Edwin van Leeuwen
wrote:
Two things that you could try:
First hitlists.byKey can be expensive (especially if hitlists
is big). Instead use:
foreach( key, value ; hitlists )
Also the filter.array.length is quite expensive. You could use
count instead.
import std.algorithm : count;
value.count!(h => h.pid >= (max_pid - max_pid_diff));
I didn't know that hitlists.byKey was that expensive, that's just
the kind of feedback I was hoping for. I'm just grasping for
straws in the online documentation when I want to do things. With
my Python background it feels as if I can still get things that
work that way.
I realize the filter.array.length thing is indeed expensive. I
find it especially horrendous that the code I've written needs to
allocate a big dynamic array that will most likely be cut down
quite drastically in this step. Unfortunately I haven't figured
out a good way to do this without storing the intermediary
results since I cannot know if there might be yet another hit for
any encountered "query" since the input file might not be sorted.
But the main reason I didn't just count the values like you
suggest is actually that I need the filtered hits in later
downstream analysis. The filtered hits for each query are used as
input to a lowest common ancestor algorithm on the taxonomic tree
(of life).
