Clone URL (Committers only): https://cms.apache.org/redirect?new=anonymous;action=diff;uri=http://jena.apache.org/documentation%2Fquery%2Ftext-query.mdtext
Chris Tomlinson Index: trunk/content/documentation/query/text-query.mdtext =================================================================== --- trunk/content/documentation/query/text-query.mdtext (revision 1816205) +++ trunk/content/documentation/query/text-query.mdtext (working copy) @@ -1,5 +1,7 @@ Title: Jena Full Text Search +Title: Jena Full Text Search + This extension to ARQ combines SPARQL and full text search via [Lucene](https://lucene.apache.org) 6.4.1 or [ElasticSearch](https://www.elastic.co) 5.2.1 (which is built on @@ -64,7 +66,20 @@ ## Table of Contents - [Architecture](#architecture) + - [External content](#external-content) + - [External applications](#external-applications) + - [Document structure](#document-structure) - [Query with SPARQL](#query-with-sparql) + - [Syntax](#syntax) + - [Input arguments](#input-arguments) + - [Output arguments](#output-arguments) + - [Query strings](#query-strings) + - [Simple queries](#simple-queries) + - [Queries with language tags](#queries-with-language-tags) + - [Queries that retrieve literals](#queries-that-retrieve-literals) + - [Queries across multiple `Field`s](#queries-across-multiple-fields) + - [Queries with _Boolean Operators_ and _Term Modifiers_](#queries-with-boolean-operators-and-term-modifiers) + - [Good practice](#good-practice) - [Configuration](#configuration) - [Text Dataset Assembler](#text-dataset-assembler) - [Configuring an analyzer](#configuring-an-analyzer) @@ -108,6 +123,7 @@ The text index uses the native query language of the index: [Lucene query language](http://lucene.apache.org/core/6_4_1/queryparser/org/apache/lucene/queryparser/classic/package-summary.html#package_description) +(with [restrictions](#input-arguments)) or [Elasticsearch query language](https://www.elastic.co/guide/en/elasticsearch/reference/5.2/query-dsl.html). @@ -134,6 +150,64 @@ By using Elasticsearch, other applications can share the text index with SPARQL search. +### Document structure + +As mentioned above, text indexing of a triple involves associating a Lucene +document with the triple. How is this done? + +Lucene documents are composed of `Field`s. Indexing and searching are performed +over the contents of these `Field`s. For an RDF triple to be indexed in Lucene the +_property_ of the triple must be +[configured in the entity map of a TextIndex](#entity-map-definition). +This associates a Lucene analyzer with the _`property`_ which will be used +for indexing and search. The _`property`_ becomes the _searchable_ Lucene +`Field` in the resulting document. + +A Lucene index includes a _default_ `Field`, which is specified in the configuration, +that is the field to search if not otherwise named in the query. In jena-text +this field is configured via the `text:defaultField` property which is then mapped +to a specific RDF property via `text:predicate` (see [entity map](#entity-map-definition) +below). + +There are several additional `Field`s that will be included in the +document that is passed to the Lucene `IndexWriter` depending on the +configuration options that are used. These additional fields are used to +manage the interface between Jena and Lucene and are not generally +searchable per se. + +The most important of these additional `Field`s is the `text:entityField`. +This configuration property defines the name of the `Field` that will contain +the _URI_ or _blank node id_ of the _subject_ of the triple being indexed. This property does +not have a default and must be specified for most uses of `jena-text`. This +`Field` is often given the name, `uri`, in examples. It is via this `Field` +that `?s` is bound in a typical use such as: + + select ?s + where { + ?s text:query "some text" + } + +Other `Field`s that may be configured: `text:uidField`, `text:graphField`, +and so on are discussed below. + +Given the triple: + + ex:SomeOne skos:prefLabel "zorn protégé a prés"@fr ; + +The following is an abbreviated illustration a Lucene document that Jena will create and +request Lucene to index: + + Document< + <uri:http://example.org/SomeOne> + <graph:urn:x-arq:DefaultGraphNode> + <label:zorn protégé a prés> + <lang:fr> + <uid:28959d0130121b51e1459a95bdac2e04f96efa2e6518ff3c090dfa7a1e6dcf00> + > + +It may be instructive to refer back to this example when considering the various +points below. + ## Query with SPARQL The URI of the text extension property function is @@ -143,63 +217,233 @@ ... text:query ... +### Syntax The following forms are all legal: - ?s text:query 'word' # query - ?s text:query (rdfs:label 'word') # query specific property if multiple - ?s text:query ('word' 10) # with limit on results - (?s ?score) text:query 'word' # query capturing also the score - (?s ?score ?literal) text:query 'word' # ... and original literal value + ?s text:query 'word' # query + ?s text:query ('word' 10) # with limit on results + ?s text:query (rdfs:label 'word') # query specific property if multiple + ?s text:query (rdfs:label 'protégé' 'lang:fr') # restrict search to French + (?s ?score) text:query 'word' # query capturing also the score + (?s ?score ?literal) text:query 'word' # ... and original literal value The most general form is: - (?s ?score ?literal) text:query (property 'query string' limit) + (?s ?score ?literal) text:query (property 'query string' limit 'lang:xx') -Only the query string is required, and if it is the only argument the -surrounding `( )` can be omitted. +#### Input arguments: -Input arguments: - | Argument | Definition | |-------------------|--------------------------------| | property | (optional) URI (including prefix name form) | -| query string | The native query string | +| query string | Lucene query string fragment | | limit | (optional) `int` limit on the number of results | +| lang:xx | (optional) language tag spec | -Output arguments: +The `property` URI is only necessary if multiple properties have been +indexed and the property being searched over is not the [default field +of the index](#entity-map-definition). +The `query string` syntax conforms the underlying index [Lucene](http://lucene.apache.org/core/6_4_1/queryparser/org/apache/lucene/queryparser/classic/package-summary.html#package_description) +or +[Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/5.2/query-dsl.html). In the case of Lucene the syntax is restricted to `Terms`, `Term modifiers`, `Boolean Operators` applied to `Terms`, and `Grouping` of terms. No use of `Fields` within the `query string` is supported. + +The optional `limit` indicates the maximum hits to be returned by Lucene. + +The `lang:xx` specification is an optional string, where _xx_ is +a BCP-47 language tag. This restricts searches to field values that were originally +indexed with the tag _xx_. Searches may be restricted to field values with no +language tag via `"lang:none"`. + +If [multilingual support](#linguistic-support-with-lucene-index) has been configured +and the `lang:xx` is used then the `property` URI must be supplied +in order for searches to work as expected. + +If both `limit` and `lang:xx` are present, then `limit` must precede +`lang:xx`. + +If only the query string is required, the surrounding `( )` can be omitted. + +#### Output arguments: + | Argument | Definition | |-------------------|--------------------------------| -| indexed term | The indexed RDF term. | +| subject URI | The subject of the indexed RDF triple. | | score | (optional) The score for the match. | -| hit | (optional) The literal matched. | +| literal | (optional) The matched object literal. | -The `property` URI is only necessary if multiple properties have been -indexed and the property being searched over is not the [default field -of the index](#entity-map-definition). Also the `property` URI **must -not** be used when the `query string` refers explicitly to one or more -fields. - -The results include the subject URI, `?s`; the `?score` assigned by the -text search engine; and the entire matched `?literal` (if the index has +The results include the _subject URI_; the _score_ assigned by the +text search engine; and the entire matched _literal_ (if the index has been [configured to store literal values](#text-dataset-assembler)). +The _subject URI_ may be a variable, e.g., `?s`, or a _URI_. In the +latter case the search is restricted to triples with the specified +subject. The _score_ and the _literal_ **must** be variables. -If the `query string` refers to more than one field, e.g., +If only the _subject_ variable, `?s`, or specific _`URI`_ is needed +then it must be written without surrounding `( )`; otherwise, an error +is signalled. - "label: printer AND description: \"large capacity cartridge\"" +### Query strings -then the `?literal` in the results will not be bound since there is no -single field that contains the match – the match is separated over -two fields. +There are several points that need to be considered when formulating +SPARQL queries using the Lucene interface. As mentioned above, in the case of Lucene the `query string` syntax is restricted to `Terms`, `Term modifiers`, `Boolean Operators` applied to `Terms`, and `Grouping` of terms. -If an output indexed term is already a known value, either as a constant -in the query or variable already set, then the index lookup becomes a -check that this is a match for the input arguments. +**No use of `Fields` within the `query string` is supported.** +#### Simple queries + +The simplest use of the jena-text Lucene integration is: + + ?s text:query "some phrase" + +This will bind `?s` to each entity URI that is the subject of a triple +that has the default property and an object literal that matches +the argument string, e.g.: + + ex:AnEntity skos:prefLabel "this is some phrase to match" + +This query form will indicate the subjects that have literals that match +without providing any information about the specific literals that matched. +If this use case is sufficient for your needs you can skip on to the +[sections on configuration](#configuration). + +#### Queries with language tags + +When working with `rdf:langString`s It may be tempting to write: + + ?s text:query "protégé"@fr + +However, the above will silently fail to return results since the +`query string` must be a simple `xsd:string` not an `rdf:langString`. + +The effective form of the above query is expressed: + + ?s text:query (skos:prefLabel "protégé" 'lang:fr') + +if the intent is to search only labels with French content. + +Even if the default _property_ is `skos:prefLabel` it is necessary +to use the above form, when [multilingual support](#linguistic-support-with-lucene-index) has been configured, rather than omitting the `property` argument +when restricting the Lucene search to a specific `lang:xx`; otherwise, +again there will be no results. + +If all one is interested in are _subjects_ with `skos:prefLabel` where +that is the `text:predicate` of the `text:defaultField` and without regard +for specified `lang:xx`s then: + + ?s text:query "protégé" + +will do the job. + +For a non-default `Field` with no language restriction, the pattern: + + ?s text:query (rdfs:label "protégé") + +may be used (see [below](#entity-map-definition) for how RDF _property_ names +are mapped to Lucene `Field` names). + +#### Queries that retrieve literals + +It is possible to retrieve the *literal*s that Lucene finds matches for +assuming that + + <#TextIndex#> text:storeValues true ; + +has been specified in the `TextIndex` configuration. So + + (?s ?sc ?lit) text:query (rdfs:label "protégé") + +will bind the matching literals to `?lit`, e.g., + + "zorn protégé a prés"@fr + +#### Queries across multiple `Field`s + +It has been mentioned earlier that the text index uses the +[native Lucene query language](http://lucene.apache.org/core/6_4_1/queryparser/org/apache/lucene/queryparser/classic/package-summary.html#package_description); +however, there are important constraints on how the Lucene query language is used within jena-text. As has been mentioned, references to Lucene `Fields` with the `query string` **are not** supported. So how are Lucene queries that would otherwise refer to multiple `Fields` expressed? + +The key is understanding that each triple is a separate document and so queries across Lucene `Fields` need to be expressed as SPARQL queries referring to the corresponding RDF _properties_. + +For example, consider the following triples: + + ex:SomePrinter + rdfs:label "laser printer" ; + ex:description "includes a large capacity cartridge" . + + assuming an appropriate configuration we might expect to retrieve `ex:SomePrinter` + with the following Lucene `query string`: + + ?s text:query "label:printer AND description:\"large capacity cartridge\"" + +This query can not find the expected results since the `AND` is interpreted +by Lucene to indicate that all documents that contain a matching `label` field _and_ +a matching `description` field are to be returned; yet, from the discussion above +regarding the [structure of Lucene documents in jena-text](#document-structure) it +is evident that there is not one but rather in fact two separate documents one with a +`label` field and one with a `description` field so an effective SPARQL query is: + + ?s text:query (rdfs:label "printer") . + ?s text:query (ex:description "large capacity cartridge") . + +which leads to `?s` being bound to `ex:SomePrinter`. + +In other words when a query is to involve two or more _properties_/`Field`s then it +expressed at the SPARQL level, as it were, versus in Lucene's query language. + +It is worth noting that the equivalent of a Lucene `OR` of `Fields` is expressed +simply via SPARQL `union`: + + { ?s text:query (rdfs:label "printer") . } + union + { ?s text:query (ex:description "large capacity cartridge") . } + +Suppose the matching literals are required for the above then it should be clear +from the above that: + + (?s ?sc1 ?lit1) text:query (skos:prefLabel "printer") . + (?s ?sc2 ?lit2) text:query (ex:description "large capacity cartridge") . + +will be the appropriate form to retrieve the _subject_ and the associated literals, `?lit1` and `?lit2`. (Obviously, in general, the _score_ variables, `?sc1` and `?sc2` +must be distinct since it is very unlikely that the scores of the two Lucene queries +will ever match). + +There is no loss of expressiveness of the Lucene query language versus the jena-text +integration of Lucene. Any cross-field `AND`s are replaced by concurrent SPARQL calls to +text:query as illustrated above and uses of Lucene `OR` can be converted to SPARQL +`union`s. Uses of Lucene `NOT` are converted to appropriate SPARQL `filter`s. + +#### Queries with _Boolean Operators_ and _Term Modifiers_ + +On the other hand the various features of the [Lucene query language](http://lucene.apache.org/core/6_4_1/queryparser/org/apache/lucene/queryparser/classic/package-summary.html#package_description) +are all available to be used for searches within a `Field`. For example, _Boolean Operators_ on _Terms_: + + ?s text:query (ex:description "(large AND cartridge)") + +and + + (?s ?sc ?lit) text:query (ex:description "(includes AND (large OR capacity))") + +or _fuzzy_ searches: + + ?s text:query (ex:description "include~") + +and so on will work as expected. + +**The key is to always surround the field query with `( )`s.** + + ### Good practice -The query engine does not have information about the selectivity of the +From the above it should be clear that best practice, except in the simplest cases +is to use explicit `text:query` forms such as: + + (?s ?sc ?lit) text:query (ex:someProperty "a single Field query") + +possibly with _limit_ and `lang:xx` arguments. + +Further, the query engine does not have information about the selectivity of the text index and so effective query plans cannot be determined programmatically. It is helpful to be aware of the following two general query patterns. @@ -394,7 +638,7 @@ is returned on a match. The value of the property is arbitrary so long as it is unique among the defined names. -#### Automatic document deletion +#### UID Field and automatic document deletion When the `text:uidField` is defined in the `EntityMap` then dropping a triple will result in the corresponding document, if any, being deleted from the text index. The value, `"uid"`, is arbitrary @@ -470,7 +714,7 @@ Support for the new `LocalizedAnalyzer` has been introduced in Jena 3.0.0 to deal with Lucene language specific analyzers. See [Linguistic Support with -Lucene Index](#linguistic-support-with-lucene-index) part for details. +Lucene Index](#linguistic-support-with-lucene-index) for details. Support for `GenericAnalyzer`s has been introduced in Jena 3.4.0 to allow the use of Analyzers that do not have built-in support, e.g., `BrazilianAnalyzer`; @@ -613,9 +857,9 @@ #### Explicit Language Field in the Index -Literals' languages of triples can be stored (during triple addition phase) into the -index to extend query capabilities. -For that, the new `text:langField` property must be set in the EntityMap assembler : +The language tag for object literals of triples can be stored (during triple addition phase) +into the index to extend query capabilities. +For that, the `text:langField` property must be set in the EntityMap assembler : <#entMap> a text:EntityMap ; text:entityField "uri" ; @@ -632,7 +876,7 @@ #### SPARQL Linguistic Clause Forms -Once the `langField` is set, you can use it directly inside SPARQL queries, for that the `'lang:xx'` +Once the `langField` is set, you can use it directly inside SPARQL queries, for that the `lang:xx` argument allows you to target specific localized values. For example: //target english literals @@ -714,7 +958,7 @@ Hence, the result set of the query will contain "institute" related subjects (institution, institutional,...) in French and in English. -**Note**: If the `text:langField` property is not set, the `text:langField` will default to"lang". +**Note**: If the `text:langField` property is not set, the `text:langField` will default to "lang". ### Generic and Defined Analyzer Support
