I restarted working on my personal search engine. It used to be called culturia [0] with too many planned features. At some point, I called it asylum [1] and focused on personal knowledge base aspects and the last iteration was called gotofish [2]
[0] https://framagit.org/a-guile-mind/culturia [1] https://framagit.org/a-guile-mind/culturia.next [2] https://git.sr.ht/~amz3/guile-gotofish I learned much from all this projects. In particular, I learned that it will be a long long long project, even if I focus only on "personal search engine" line of work. The last iteration, gotofish, was not too bad even if it has bitrot. Based on my research and practical experiment, it seems very clear that there is no workaround the use of map-reduce, that might be known as n-par-for-each [3]. [3] https://www.gnu.org/software/guile/manual/html_node/Parallel-Forms.html#index-n_002dpar_002dfor_002deach I made a prototype similar to that n-par-for-each, except it works with guile-fibers, is asynchronous and works with a shared pool of threads instead of spawning N threads for each incoming query like gotofish does. Related blog post: https://hyper.dev/blog/on-the-road-to-babelia.html If you want to help or discuss those matters, do not hesitate to reply to this message. Cheers, Amirouche ~ amz3 ~ https://hyper.dev