Clone URL (Committers only): https://cms.apache.org/redirect?new=anonymous;action=diff;uri=http://jena.apache.org/documentation%2Fquery%2Ftext-query.mdtext
Chris Tomlinson Index: trunk/content/documentation/query/text-query.mdtext =================================================================== --- trunk/content/documentation/query/text-query.mdtext (revision 1815762) +++ trunk/content/documentation/query/text-query.mdtext (working copy) @@ -1,5 +1,7 @@ Title: Jena Full Text Search +Title: Jena Full Text Search + This extension to ARQ combines SPARQL and full text search via [Lucene](https://lucene.apache.org) 6.4.1 or [ElasticSearch](https://www.elastic.co) 5.2.1 (which is built on @@ -64,7 +66,20 @@ ## Table of Contents - [Architecture](#architecture) + - [External content](#external-content) + - [External applications](#external-applications) + - [Document structure](#document-structure) - [Query with SPARQL](#query-with-sparql) + - [Syntax](#syntax) + - [Input arguments](#input-arguments) + - [Output arguments](#output-arguments) + - [Query strings](#query-strings) + - [Simple queries](#simple-queries) + - [Queries with language tags](#queries-with-language-tags) + - [Queries that retrieve literals](#queries-that-retrieve-literals) + - [Queries across multiple `Field`s](#queries-across-multiple-fields) + - [Queries within a `Field`](#queries-within-a-field) + - [Good practice](#good-practice) - [Configuration](#configuration) - [Text Dataset Assembler](#text-dataset-assembler) - [Configuring an analyzer](#configuring-an-analyzer) @@ -134,6 +149,69 @@ By using Elasticsearch, other applications can share the text index with SPARQL search. +### Document structure + +As mentioned above, text indexing of a triple involves associating a Lucene +document with the triple. How is this done? + +Lucene documents are composed of `Field`s. Indexing and searching are performed +over the contents of these `Field`s. For an RDF triple to be indexed in Lucene the +_property_ of the triple must be +[configured in the entity map of a TextIndex](#entity-map-definition). +This associates a Lucene analyzer with the _`property`_ which will be used +for indexing and search. The _`property`_ becomes the _searchable_ Lucene +`Field` in the resulting document. + +A Lucene index includes a _default_ `Field`, which is specified in the configuration, +that is the field to search if not otherwise named in the query. In jena-text +this field is configured via the `text:defaultField` property which is then mapped +to a specific RDF property via `text:predicate` (see [entity map](#entity-map-definition) +below). + +There are several additional `Field`s that will be included in the +document that is passed to the Lucene `IndexWriter` depending on the +configuration options that are used. These additional fields are used to +manage the interface between Jena and Lucene and are not generally +searchable per se. + +The most important of these additional `Field`s is the `text:entityField`. +This configuration property defines the name of the `Field` that will contain +the _URI_ or _blank node id_ of the _subject_ of the triple being indexed. This property does +not have a default and must be specified for most uses of `jena-text`. This +`Field` is often given the name, `uri`, in examples. It is via this `Field` +that `?s` is bound in a typical use such as: + + select ?s + where { + ?s text:query "some text" + } + +Other `Field`s that may be configured: `text:uidField`, `text:graphField`, +and so on are discussed below. + +Given the triple: + + ex:SomeOne skos:prefLabel "zorn protégé a prés"@fr ; + +The following illustrates a Lucene document that Jena will create and +request Lucene to index: + + Document< + stored, indexed, indexOptions=DOCS <uri:http://example.org/SomeOne> + indexed, omitNorms, indexOptions=DOCS <graph:urn:x-arq:DefaultGraphNode> + stored, indexed, tokenized <label:zorn protégé a prés> + stored, indexed, omitNorms, indexOptions=DOCS <lang:fr> + stored, indexed, tokenized <label_fr:zorn protégé a prés> + stored, indexed, omitNorms, indexOptions=DOCS <uid:28959d0130121b51e1459a95bdac2e04f96efa2e6518ff3c090dfa7a1e6dcf00> + stored, indexed, tokenized <graph:urn:x-arq:DefaultGraphNode> + stored, indexed, omitNorms, indexOptions=DOCS <lang:fr> + stored, indexed, tokenized <graph_fr:urn:x-arq:DefaultGraphNode> + stored, indexed, omitNorms, indexOptions=DOCS <uid:b668c6cb80475194a2ce3087c15afbcb4baf6150da98e95b590ceedd10cf242f> + > + +It may be instructive to refer back to this example when considering the various +points below. + ## Query with SPARQL The URI of the text extension property function is @@ -143,63 +221,248 @@ ... text:query ... +### Syntax The following forms are all legal: - ?s text:query 'word' # query - ?s text:query (rdfs:label 'word') # query specific property if multiple - ?s text:query ('word' 10) # with limit on results - (?s ?score) text:query 'word' # query capturing also the score - (?s ?score ?literal) text:query 'word' # ... and original literal value + ?s text:query 'word' # query + ?s text:query ('word' 10) # with limit on results + ?s text:query (rdfs:label 'word') # query specific property if multiple + ?s text:query (rdfs:label 'protégé' 'lang:fr') # restrict search to French + (?s ?score) text:query 'word' # query capturing also the score + (?s ?score ?literal) text:query 'word' # ... and original literal value The most general form is: - (?s ?score ?literal) text:query (property 'query string' limit) + (?s ?score ?literal) text:query (property 'query string' limit 'lang:xx') -Only the query string is required, and if it is the only argument the -surrounding `( )` can be omitted. +#### Input arguments: -Input arguments: - | Argument | Definition | |-------------------|--------------------------------| | property | (optional) URI (including prefix name form) | -| query string | The native query string | +| query string | Lucene query string fragment | | limit | (optional) `int` limit on the number of results | +| lang:xx | (optional) language tag spec | -Output arguments: +The `property` URI is only necessary if multiple properties have been +indexed and the property being searched over is not the [default field +of the index](#entity-map-definition). Also the `property` URI **must +not** be used when the `query string` refers explicitly to one or more +fields. The optional `limit` indicates the maximum hits to be returned by Lucene. +The `lang:xx` specification is an optional string, where _xx_ is +a BCP-47 language tag. This restricts searches to field values that were originally +indexed with the tag _xx_. Searches may be restricted to field values with no +language tag via `"lang:none"`. The use of the `lang:xx` is only effective if +[multilingual support](#linguistic-support-with-lucene-index) has been configured. +Further, if the `lang:xx` is used then the `property` URI must be supplied +in order for searches to work. + +If both `limit` and `lang:xx` are present, then `limit` must precede +`lang:xx`. + +If only the query string is required, the surrounding `( )` can be omitted. + +#### Output arguments: + | Argument | Definition | |-------------------|--------------------------------| -| indexed term | The indexed RDF term. | +| subject URI | The subject of the indexed RDF triple. | | score | (optional) The score for the match. | -| hit | (optional) The literal matched. | +| literal | (optional) The matched object literal. | -The `property` URI is only necessary if multiple properties have been -indexed and the property being searched over is not the [default field -of the index](#entity-map-definition). Also the `property` URI **must -not** be used when the `query string` refers explicitly to one or more +The results include the _subject URI_; the _score_ assigned by the +text search engine; and the entire matched _literal_ (if the index has +been [configured to store literal values](#text-dataset-assembler)). +The _subject URI_ may be a variable, e.g., `?s`, or a _URI_. In the +latter case the search is restricted to triples with the specified +subject. The _score_ and the _literal_ **must** be variables. + +If only the _subject_ variable, `?s`, or specific _`URI`_ is needed +then it must be written without surrounding `( )`; otherwise, an error +is signalled. + +### Query strings + +There are several points that need to be considered when formulating +SPARQL queries using the Lucene interface. + +#### Simple queries + +The simplest use of the jena-text Lucene integration is: + + ?s text:query "some phrase" + +This will bind `?s` to each entity URI that is the subject of a triple +that has the default property and an object literal that matches +the argument string, e.g.: + + ex:AnEntity skos:prefLabel "this is some phrase to match" + +This query form will indicate the subjects that have literals that match +without providing any information about the specific literals that matched. +If this use case is sufficient for your needs you can skip on to the +[sections on configuration](#configuration). + +#### Queries with language tags + +When working with `rdf:langString`s It may be tempting to write: + + ?s text:query "protégé"@fr + +However, the above will silently fail to return results since the +`query string` must be a simple `xsd:string` not an `rdf:langString`. + +The effective form of the above query is expressed: + + ?s text:query (skos:prefLabel "protégé" 'lang:fr') + +if the intent is to search only labels with French content. + +Even if the default _property_ is `skos:prefLabel` it is necessary +to use the above form rather than omitting the `property` argument +when restricting the Lucene search to a specific `lang:xx`; otherwise, +again there will be no results. + +If all one is interested in are _subjects_ with `skos:prefLabel` where +that is the `text:predicate` of the `text:defaultField` and without regard +for specified `lang:xx`s then: + + ?s text:query "protégé" + +will do the job. + +For a non-default `Field` with no language restriction, the patterns: + + ?s text:query (rdfs:label "protégé") + +or + + ?s text:query "rdfsLabel:protégé" + +may be used (see [below](#entity-map-definition) for how RDF _property_ names +are mapped to Lucene `Field` names). However, as mentioned earlier, + + ?s text:query ("rdfsLabel:protégé" "lang:fr") + +will result in an error owing to the way in which the jena-text composes the +query string to Lucene in the presence of the `"lang:fr"` argument. + +#### Queries that retrieve literals + +It is possible to retrieve the *literal*s that Lucene finds matches for +assuming that + + <#TextIndex#> text:storeValues true ; + +has been specified in the `TextIndex` configuration. So + + (?s ?sc ?lit) text:query (rdfs:label "protégé") + +will bind the matching literals to `?lit`, e.g., + + "zorn protégé a prés"@fr + +However, it is important to note that the apparently equivalent form: + + (?s ?sc ?lit) text:query "rdfsLabel:protégé" + +will fail to produce a binding for `?lit` even though `?s` and `?sc` are +bound as expected. + +So if the _literal_ matches are needed you **must use** the query arguments that +list the _property_ explicitly, except in the simple case of a query against +the default `Field`/_property_. + +#### Queries across multiple `Field`s + +It has been mentioned earlier that the text index uses the +[native Lucene query language](http://lucene.apache.org/core/6_4_1/queryparser/org/apache/lucene/queryparser/classic/package-summary.html#package_description); +however, there are important constraints on how the Lucene query language is used within jena-text. +This is owing to the fact that jena-text composes the query string that is sent to Lucene so that +features such as `lang:xx` may be implemented. Other aspects of using the Lucene query language +reflect the fact that each triple is a separate document. + +This latter observation is important when considering queries that are intended to involve several fields. -The results include the subject URI, `?s`; the `?score` assigned by the -text search engine; and the entire matched `?literal` (if the index has -been [configured to store literal values](#text-dataset-assembler)). +For example, consider the following triples: -If the `query string` refers to more than one field, e.g., + ex:SomePrinter + rdfs:label "laser printer" ; + ex:description "includes a large capacity cartridge" . - "label: printer AND description: \"large capacity cartridge\"" + assuming an appropriate configuration we might expect to retrieve `ex:SomePrinter` + with the following query: -then the `?literal` in the results will not be bound since there is no -single field that contains the match – the match is separated over -two fields. + ?s text:query "label:printer AND description:\"large capacity cartridge\"" -If an output indexed term is already a known value, either as a constant -in the query or variable already set, then the index lookup becomes a -check that this is a match for the input arguments. +However, this query will fail to find the expected results since the `AND` is interpreted +by Lucene to indicate that all documents that contain a matching `label` field _and_ +a matching `description` field are to be returned. Yet from the discussion above +regarding the [structure of Lucene documents in jena-text](#document-structure) it +is evident that there is not one but rather in fact two separate documents one with a +`label` field and one with a `description` field so an effective query is: + ?s text:query "label:printer" . + ?s text:query "description:\"large capacity cartridge\"" . + +which leads to `?s` being bound to `ex:SomePrinter`. + +In other words when a query is to involve two or more _properties_/`Field`s then it +expressed at the SPARQL level, as it were, versus in Lucene's query language. + +It is worth noting that: + + ?s text:query "label:printer OR description:\"large capacity cartridge\"" + +works simply because Lucene is required to do nothing more than a _union_ of +matching documents the same as if written: + + { ?s text:query "label:printer" . } + union + { ?s text:query "description:\"large capacity cartridge\"" . } + +Suppose the matching literals are required for the above then it should be clear +from the above that: + + (?s ?sc1 ?lit1) text:query (skos:prefLabel "printer") . + (?s ?sc2 ?lit2) text:query (ex:description "large capacity cartridge") . + +will be the appropriate form to retrieve the _subject_ and the associated literals. + +There is no loss of expressiveness of the Lucene query language versus the jena-text +integration of Lucene. Any cross-field `AND`s are replaced by concurrent SPARQL calls to +text:query as illustrated above and uses of Lucene `OR` can be converted to SPARQL +`union`s. Uses of Lucene `NOT` are converted to appropriate SPARQL `filter`s. + +#### Queries within a `Field` + +On the other hand the various features of the [Lucene query language](http://lucene.apache.org/core/6_4_1/queryparser/org/apache/lucene/queryparser/classic/package-summary.html#package_description) +are all available to be used for searches within a `Field`. For example: + + ?s text:query "description:(large AND cartridge)" + +and + + (?s ?sc ?lit) text:query (ex:description "(includes AND (large OR capacity))") + +will work as expected. + +The key is to always surround the field query with `( )`s. + + ### Good practice -The query engine does not have information about the selectivity of the +From the above it should be clear that best practice, except in the simplest cases +is to use explicit `text:query` forms such as: + + (?s ?sc ?lit) text:query (ex:someProperty "a single Field query") + +possibly with _limit_ and `lang:xx` arguments. + +Further, the query engine does not have information about the selectivity of the text index and so effective query plans cannot be determined programmatically. It is helpful to be aware of the following two general query patterns. @@ -394,7 +657,7 @@ is returned on a match. The value of the property is arbitrary so long as it is unique among the defined names. -#### Automatic document deletion +#### UID Field and automatic document deletion When the `text:uidField` is defined in the `EntityMap` then dropping a triple will result in the corresponding document, if any, being deleted from the text index. The value, `"uid"`, is arbitrary @@ -632,7 +895,7 @@ #### SPARQL Linguistic Clause Forms -Once the `langField` is set, you can use it directly inside SPARQL queries, for that the `'lang:xx'` +Once the `langField` is set, you can use it directly inside SPARQL queries, for that the `lang:xx` argument allows you to target specific localized values. For example: //target english literals @@ -714,7 +977,7 @@ Hence, the result set of the query will contain "institute" related subjects (institution, institutional,...) in French and in English. -**Note**: If the `text:langField` property is not set, the `text:langField` will default to"lang". +**Note**: If the `text:langField` property is not set, the `text:langField` will default to "lang". ### Generic and Defined Analyzer Support
