Hi, On Tue, 9 Mar 2021 at 14:37, Taylan Kammer <taylan.kam...@gmail.com> wrote:
> This discussion made me realize that "guix search" might benefit from > the following improvement though: I think the relevance score for a > search result should be increased significantly if the searched word is > a standalone (not substring) part of a package's name when the name is > split into dash-separated words. Currently, perfect match uses the weight of 5 and substring match uses 1. You are proposing to add something between, say 3, for perfect match on substring delimited by dash. Why not. > For instance, the package "emacs-hl-todo" should get a much higher score > than "emacs-mastodon" when searching for "todo". Currently the Mastodon > one has score 11 and the todo one only 9. Here how the relevance score reads: query: todo | field | emacs-hl-todo | emacs-mastodon | weight | |-------------+----------------+-----------------+--------| | name | 1 | 1 | 4 | | synopsis | 1 | 1 | 3 | | description | 1 | 2 | 2 | |-------------+----------------+-----------------+--------| | total | 1*4+1*3+2*1= 9 | 1*4+1*3+2*2= 11 | | Therefore, something looks wrong here: the score for emacs-hl-todo should be 1*4+1*5*3+1*5*2= 29 because the term TODO should be considered as a perfect match for the query todo. > The same thing goes for the synopsis and description of the package, but > with respectively lower increases to the score. (I.e. name > synopsis > > description.) Your proposal just needs the tweak of 'score' in the function 'relevance' from (guix ui). The weight for the field is another part (see %package-metrics in (guix ui)) > Handling of plurals like "todos" instead of "todo" would also be great > but could be left to a later step. The issue with this is that it is strongly connected to the language. Therefore, an external library implementing Natural Language should be added. And I am not convinced it is worth at the CLI level. > Any thoughts about / objections to this idea? To be honest I haven't > checked if there's maybe already a bug report about this. If you are interested, there is such discussion in this heavy thread: <http://issues.guix.gnu.org/issue/39258> And the 'relevance' function could be improved, for sure. For example, I proposed TF-IDF here: <https://lists.gnu.org/archive/html/guix-devel/2019-07/msg00252.html> and I did some tiny math calculs (optimization) to compute "better" relevance weight (%package-metrics) but the current choice are not so bad and simple enough. :-) Previous week, I have started to examine a strategy based on Bag-Of-Word and some word embedings strategies; mimicking a simple autoencoder [1] such as Word2Vec [2] but since the Guile tools are poor in this field, I have started to use Julia first to look if it is worth to implement or not such solution. My idea is to see how the packages cluster based on the synopsis+description information, then ideally based on this, we should be able to define package similarity and "synonyms". Well, if you are student and you are looking for a cool project about Machine Learning and Data Science, ping me. :-) 1: <https://en.wikipedia.org/wiki/Autoencoder> 2: <https://en.wikipedia.org/wiki/Word2vec> Cheers, simon