magibney commented on a change in pull request #476: URL: https://github.com/apache/solr/pull/476#discussion_r789890276
########## File path: solr/solr-ref-guide/src/dense-vector-search.adoc ########## @@ -0,0 +1,308 @@ += Dense Vector Search +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +The Apache Solr *Dense Vector Search* module adds support for indexing and searching dense numerical vectors. + +https://en.wikipedia.org/wiki/Deep_learning[Deep learning] can be used to produce a vector representation of both the query and the documents in a corpus of information. + +These neural network-based techniques are usually referred to as neural search, an industry derivation from the academic field of https://www.microsoft.com/en-us/research/uploads/prod/2017/06/fntir2018-neuralir-mitra.pdf[Neural information Retrieval]. + +== Important Concepts + +=== Dense Vector Representation +A dense vector describes information as an array of elements, each of them explicitly defined. Review comment: Thanks! I took another pass at this below. Starting from your latest revision, I partially reverted a few things to the original version I drafted (mostly just the beginning of the first paragraph). I stuck with your reformatting into more separate paragraphs, and I appreciated your reduction of my "overuse" of "quotes" :-) -- so left that as you had it. I also left the two places where you replaced "Solr" with "Apache Solr", though personally I'd be inclined to go with the more succinct "Solr"? I linked to the wikipedia page for "inverted index", analogous to what you had done for "Bag of words" -- though I'm not sure the external link is necessary. The only other change I introduced was "distills semantic meaning" => "distills _approximate_ semantic meaning". I don't feel too strongly about that, but I do think it's more accurate, and lets us humans continue to feel special :-) >=== Dense Vector Representation >A traditional tokenized https://en.wikipedia.org/wiki/Inverted_index[inverted index] can be considered to model text as a "sparse" vector, in which each term in the corpus corresponds to one vector dimension. In such a model, the number of dimensions is generally quite high (corresponding to the term dictionary cardinality), and the vector for any given document contains mostly zeros (hence it is sparse, as only a handful of terms that exist in the overall index will be present in any given document). > >Dense vector representation contrasts with term-based sparse vector representation in that it distills approximate semantic meaning into a fixed (and limited) number of dimensions. > >The number of dimensions in this approach is generally much lower than the sparse case, and the vector for any given document is dense, as most of its dimensions are populated by non-zero values. > >In contrast to the sparse approach (for which tokenizers are used to generate sparse vectors directly from text input) the task of generating vectors must be handled in application logic external to Apache Solr. > >There may be cases where it makes sense to directly search data that natively exists as a vector (e.g., scientific data); but in a text search context, it is likely that users will leverage deep learning models such as https://en.wikipedia.org/wiki/BERT_(language_model)[BERT] to encode textual information as dense vectors, supplying the resulting vectors to Apache Solr explicitly at index and query time. > >For additional information you can refer to this https://sease.io/2021/12/using-bert-to-improve-search-relevance.html[blog post]. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
