Revision: 16768
http://sourceforge.net/p/gate/code/16768
Author: valyt
Date: 2013-08-07 12:36:26 +0000 (Wed, 07 Aug 2013)
Log Message:
-----------
Added some examples of how to specify:
- more than one feature name in a list of features in the index template
- features of type string for an annotation type
- metadata helpers other than OriginalMarkupsMetadataHelper.
Modified Paths:
--------------
mimir/trunk/doc/default-index-config.txt
mimir/trunk/doc/indexing.tex
mimir/trunk/doc/mimir-guide.pdf
mimir/trunk/doc/plugins.tex
Modified: mimir/trunk/doc/default-index-config.txt
===================================================================
--- mimir/trunk/doc/default-index-config.txt 2013-08-04 15:00:29 UTC (rev
16767)
+++ mimir/trunk/doc/default-index-config.txt 2013-08-07 12:36:26 UTC (rev
16768)
@@ -2,27 +2,29 @@
/@ 2 @/import gate.mimir.SemanticAnnotationHelper.Mode
/@ 3 @/import gate.mimir.index.OriginalMarkupMetadataHelper
/@ 4 @/import gate.mimir.db.DBSemanticAnnotationHelper as DefaultHelper
-/@ 5 @/
-/@ 6 @/tokenASName = ""
-/@ 7 @/tokenAnnotationType = ANNIEConstants.TOKEN_ANNOTATION_TYPE
-/@ 8 @/tokenFeatures = {
-/@ 9 @/ string()
-/@ 10 @/ category()
-/@ 11 @/ root()
-/@ 12 @/}
-/@ 13 @/
-/@ 14 @/semanticASName = ""
-/@ 15 @/semanticAnnotations = {
-/@ 16 @/ index {
-/@ 17 @/ annotation helper:new DefaultHelper(annType:'Sentence')
-/@ 18 @/ }
-/@ 19 @/ index {
-/@ 20 @/ annotation helper:new DefaultHelper(annType:'Person',
nominalFeatures:["gender"])
-/@ 21 @/ annotation helper:new DefaultHelper(annType:'Location',
nominalFeatures:["locType"])
-/@ 22 @/ annotation helper:new DefaultHelper(annType:'Organization',
nominalFeatures:["orgType"])
-/@ 23 @/ annotation helper:new DefaultHelper(annType:'Date',
integerFeatures:["normalized"])
-/@ 24 @/ annotation helper:new DefaultHelper(annType:'Document',
integerFeatures:["date"], mode:Mode.DOCUMENT)
-/@ 25 @/ }
-/@ 26 @/}
-/@ 27 @/documentRenderer = new OriginalMarkupMetadataHelper()
-/@ 28 @/documentMetadataHelpers = [documentRenderer]
\ No newline at end of file
+/@ 5 @/import gate.mimir.util.DocumentFeaturesMetadataHelper
+/@ 6 @/
+/@ 7 @/tokenASName = ""
+/@ 8 @/tokenAnnotationType = ANNIEConstants.TOKEN_ANNOTATION_TYPE
+/@ 9 @/tokenFeatures = {
+/@ 10 @/ string()
+/@ 11 @/ category()
+/@ 12 @/ root()
+/@ 13 @/}
+/@ 14 @/
+/@ 15 @/semanticASName = ""
+/@ 16 @/semanticAnnotations = {
+/@ 17 @/ index {
+/@ 18 @/ annotation helper:new DefaultHelper(annType:'Sentence')
+/@ 19 @/ }
+/@ 20 @/ index {
+/@ 21 @/ annotation helper:new DefaultHelper(annType:'Person',
nominalFeatures:["gender", "title"], stringFeatures:["name"])
+/@ 22 @/ annotation helper:new DefaultHelper(annType:'Location',
nominalFeatures:["locType", "continent"])
+/@ 23 @/ annotation helper:new DefaultHelper(annType:'Organization',
nominalFeatures:["orgType"])
+/@ 24 @/ annotation helper:new DefaultHelper(annType:'Date',
integerFeatures:["normalized"])
+/@ 25 @/ annotation helper:new DefaultHelper(annType:'Document',
integerFeatures:["date"], mode:Mode.DOCUMENT)
+/@ 26 @/ }
+/@ 27 @/}
+/@ 28 @/documentRenderer = new OriginalMarkupMetadataHelper()
+/@ 29 @/documentFeaturesHelper = new
DocumentFeaturesMetadataHelper("date","source","id", "type")
+/@ 30 @/documentMetadataHelpers = [documentRenderer,
documentFeaturesHelper]
\ No newline at end of file
Modified: mimir/trunk/doc/indexing.tex
===================================================================
--- mimir/trunk/doc/indexing.tex 2013-08-04 15:00:29 UTC (rev 16767)
+++ mimir/trunk/doc/indexing.tex 2013-08-07 12:36:26 UTC (rev 16768)
@@ -38,13 +38,13 @@
The various sections of the template are as follows:
\subsection*{Imports}
-\defaultIndexTemplate[linerange=1-4,firstnumber=1]
+\defaultIndexTemplate[linerange=1-5,firstnumber=1]
The template can optionally start with import statements to import any Java
classes that are used further on in the template.
\subsection*{Token Definitions}\label{sec:indexing:tokens}
-\defaultIndexTemplate[linerange=6-12,firstnumber=6]
+\defaultIndexTemplate[linerange=7-13,firstnumber=7]
The next section of the template deals with the {\em tokens} that \Mimir\ will
base its index on. \Mimir\ sees every document as a stream of tokens rather
@@ -84,15 +84,14 @@
relevant \lstinline!import! statement at the top.
\subsection*{Semantic Annotations}\label{sec:indexing:helpers}
-\defaultIndexTemplate[linerange={14-18},firstnumber=14]
+\defaultIndexTemplate[linerange={15-19},firstnumber=15]
The next section defines the {\em semantic annotations} that \Mimir\ will
include in the index. Each \lstinline!index! block defines a set of semantic
annotation types that will be indexed and stored together in one index. The
choice of how to group annotation types together into indexes can affect the
indexing speed, as the annotations within one index are processed sequentially
-by a single thread, whereas types in separate indexes can be indexed in
-parallel.
+by a single thread, whereas types in separate indexes are indexed in parallel.
Each annotation type to be indexed is introduced by ``{\tt annotation}''. This
is a method call in the Groovy DSL which takes the following named arguments:
@@ -118,21 +117,21 @@
types, which can be configured to handle any annotation type and features, and
{\em special-purpose} helpers that are designed to handle specific annotation
types. \Mimir\ supplies two generic helper implementations in the {\tt db-h2}
-and {\tt sesame} plugins\footnote{A third implementation in the {\tt ordi}
-plugin is now deprecated. This stores the data in the same underlying OWLIM
-semantic repository format but accesses it through a different API
-abstraction.} that store annotation information in a relational database and a
-knowledge base respectively. For the most standard cases, one or other of
-these default helper implementations should be sufficient. One sample
+and {\tt sesame} plugins that store annotation information in a relational
+database and a knowledge base respectively. For the most standard cases, one
or
+other of these default helper implementations should be sufficient. One sample
special-purpose helper for {\tt Measurement} annotations (as generated by the
-GATE {\tt Tagger\_Measurements} plugin) is also provided, in the
-{\tt measurements} plugin. This is intended both to be useful in its own right
-and to serve as a template for how to implement your own helpers for other
-complex annotation types. The {\tt sparql} plugin provides a helper that can
-wrap any other helper and add the ability to query for URI-valued features by
-making a query to a SPARQL endpoint. These plugins are discussed in detail in
-chapter~\ref{sec:plugins}.
+GATE {\tt Tagger\_Measurements} plugin) is also provided, in the {\tt
+measurements} plugin. This is intended both to be useful in its own right and
+to serve as a template for how to implement your own helpers for other complex
+annotation types. The {\tt sparql} plugin provides a helper that can wrap any
+other helper and add the ability to query for URI-valued features by making a
+query to a SPARQL endpoint.
+The plugins that include all the provided semantic annotation helpers are
+discussed in detail in Chapter~\ref{sec:plugins}. Detailed documentation for
+configuring each of the helpers is available there.
+
{\bf Note for users upgrading from \Mimir\ 3.2.0 and earlier:} the previous
index template DSL style using the annotation type as the method name and the
nominalFeatures etc. as parameters is still supported but should be considered
@@ -143,7 +142,7 @@
Starting with \Mimir\ version 3.4.0, annotation helpers can also be used to
index document features.
\clearpage
-\defaultIndexTemplate[linerange={24-24},firstnumber=24]
+\defaultIndexTemplate[linerange={25-25},firstnumber=25]
The above declaration (note the {\tt mode} parameter!) creates a new Semantic
Annotation Helper that uses the document features instead of the features from
@@ -171,7 +170,7 @@
\end{itemize}
\subsection*{Document Rendering and Metadata}
-\defaultIndexTemplate[linerange={27-28},firstnumber=27]
+\defaultIndexTemplate[linerange={28-30},firstnumber=28]
The final part of the template concerns how document-level metadata is indexed,
and how this can be combined with the document text to render the document
@@ -190,6 +189,21 @@
order to function correctly. Other metadata helpers may be added to the list
if required.
+In the listing above, we use one instance of
+\lstinline!OriginalMarkupMetadataHelper! as a document renderer. To enable it
to
+fucntion, we also include the same object instance in the list of metadata
+helpers. Additionally, we also construct an instance of
+\lstinline!DocumentFeaturesMetadataHelper!, which we name
+\lstinline!documentFeaturesHelper!, and we then add to the list of metadata
+helpers. \lstinline!DocumentFeaturesMetadataHelper! instances can be used to
+store additional metadata in the index being constructed. The metadata values
to
+be stored must be provided in the form of GATE document features on the
+documents being indexed. When such values are present at indexing time, they
are
+serialised and stored in the index. At search time, the stored metadata fields
+can be retrieved back from the index. Note that the values used must be
+serialisable to be usable (i.e. they must implement the
+\lstinline!java.io.Serializable!) interface.
+
\subsection*{Direct Indexes}
\label{sec:direct-indexes}
Starting with version 5, \Mimir{} can build direct indexes as well as inverted
Modified: mimir/trunk/doc/mimir-guide.pdf
===================================================================
(Binary files differ)
Modified: mimir/trunk/doc/plugins.tex
===================================================================
--- mimir/trunk/doc/plugins.tex 2013-08-04 15:00:29 UTC (rev 16767)
+++ mimir/trunk/doc/plugins.tex 2013-08-07 12:36:26 UTC (rev 16768)
@@ -25,8 +25,8 @@
// \ldots
semanticAnnotations = {
index {
- annotation helper:new DefaultHelper(annType:'Person',
nominalFeatures:["gender"])
- }
+ annotation helper:new DefaultHelper(annType:'Person',
+ nominalFeatures:["gender", "title"], stringFeatures:["name"])) }
}
\end{lstlisting}
This was sent by the SourceForge.net collaborative development platform, the
world's largest Open Source development site.
------------------------------------------------------------------------------
Get 100% visibility into Java/.NET code with AppDynamics Lite!
It's a free troubleshooting tool designed for production.
Get down to code-level detail for bottlenecks, with <2% overhead.
Download for free and get started troubleshooting in minutes.
http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
_______________________________________________
GATE-cvs mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/gate-cvs
