Two cents: It follows convention well: W3C XML namespaces share "prefix:localName" so one could mix without colliding. Java's package naming does the same (for classes with owner-scoped names).
The toolkit ships live under "opennlp:", while application keys can never collide with ours and this gives attribution to the feature data added. I've done the dogfood test (PoCs on glossary matching, PII masking, coreference chains,etc ).. doesn't slow things down and easy to code against. -kr So I'm a fan because I know it works well. But we CAN change it - and open to any suggestions. On Tue, Jul 21, 2026 at 5:18 AM Richard Zowalla <[email protected]> wrote: > Any additional input? > > On 2026/07/17 13:55:56 Kristian Rickert wrote: > > Everyone: > > > > The draft PR (#1182) now reflects all 4 points, one commit per point so > > each is reviewable on its own. > > > > tl;dr - Everything promised in the last mail is now spec text plus > > enforcing code plus contract tests, nothing is prose-only. > > > > 1. Namespaced ids (86f4bee). Every key the toolkit defines carries the > > "opennlp:" prefix: opennlp:sentences, opennlp:tokens, opennlp:pos, > > opennlp:entities, and opennlp:lemmas / opennlp:stems on the adapters. > > Extensions use their own prefix, bare ids stay legal for > application-local > > layers. The rule is stated on Layers, on LayerKey, and in the manual > > chapter. > > 2. Per-key scope (6c38c23). LayerKey now declares POSITIONAL or DOCUMENT. > > Positional is the default and guarantees a span on every annotation; > > LayerKey.document(id, type) creates a key for whole-document values > > (language id, category distributions, provenance), and > Annotation.of(value) > > builds the span-less entry. The container polices it in both directions > at > > insertion, naming the layer: a span-less annotation under a positional > key > > is rejected, a spanned one under a document key likewise. One consequence > > you should know before commenting: key equality is now (id, type, scope), > > so a positional and a document key with the same id and type are distinct > > layers, same as the existing id-with-different-type rule. > > 3. Invariants (a657c00). Transcribed onto the Document interface and > into > > the manual: insertion order is preserved and never reordered, layers are > > immutable once added and detached from the caller's input list, add is > > once-only operation that rejects duplicates by naming the key, therefore > > index references stay valid for the document's lifetime. The wording > > matches what the contract tests enforce, nothing new was invented for the > > prose. > > 4. Gold versus predicted (48321ff). Documented as the convention on > Layers > > and in the manual: gold:opennlp:tokens beside opennlp:tokens, and a > > contract test pins that both live independently and neither can replace > the > > other. > > > > Thoughts are encouraged for the draft "opennlp:" prefix itself (versus > > something dotted) and the gold:opennlp:tokens double-prefix shape. I > > prefer the ":" as the "." convention feels less of a separation. > > > > This is a great moment to discuss, as it'll establish a new convention. > > > > - Kristian > > > > On Thu, Jul 16, 2026 at 1:58 PM Kristian Rickert <[email protected]> > wrote: > > > > > Hi Richard, > > > > > > Thanks for doing the survey before forming an opinion; the convergence > you > > > found is the best argument the design could get. I am adopting all four > > > points. Short version first: > > > > > > tl;dr - Spans are baked in by construction. The only span-less things > are > > > whole-document facts, and those are declared at the key level, not > skipped > > > at return time. > > > > > > 1. Key identity. Equality is the (id, type) pair, pinned by contract > > > tests: same id and type address one layer, same id with a different > type is > > > a distinct coexisting layer. Standard keys get namespaced ids before > the > > > freeze ("opennlp:" prefix, extensions use their own prefix, bare ids > stay > > > legal for application-local layers). > > > > > > 2. Annotations without spans. The span is structurally mandatory today: > > > the annotation record rejects a null span and the container rejects > > > out-of-bounds spans, so no code path returns a span-less result. The > census > > > of every annotator built so far: > > > > > > > > > > +---------------------------------+--------------------------------------+ > > > | Annotator | Span anchors on > > > | > > > > > > > +---------------------------------+--------------------------------------+ > > > | Sentence / tokenizer adapters | detected spans; tokens shifted to > > > | > > > | | document coordinates > > > | > > > | POS / lemmatizer / stemmer | each token's span > > > | > > > | Name finder | the entity span > > > | > > > | Glossary | the match span > > > | > > > | PII | the mention span > > > | > > > | Coreference (chains) | each member mention's span > > > | > > > | Money / quantity / temporal / | the mention span > > > | > > > | currency conversion | > > > | > > > | Document date | the electing mention's span > > > | > > > | Geocoding (locations) | the resolved mention's span > > > | > > > | Containment (hierarchy) | the location mention's span > > > | > > > | Dependencies | the dependent token's span > > > | > > > | Relations | the covering span of both > arguments > > > | > > > | Embeddings | the source annotation's span > > > | > > > > > > > +---------------------------------+--------------------------------------+ > > > > > > Only one layer spans the document as a whole (the region ballot, > currently > > > on a whole-document span). Proposal: scope is declared per key, never > per > > > annotation. Positional keys guarantee a span on every entry; > > > document-scoped keys carry values without spans (language id, > categories, > > > provenance, the ballot). That is the UIMA offset-bearing versus > offset-free > > > split you cited, at library weight, and document-scoped layers staying > > > lists fits the real payloads (a categorizer returns a scored > distribution). > > > > > > 3. Invariants. All three are implemented and contract-tested, and the > spec > > > will state them: insertion order is preserved and never sorted; layers > are > > > immutable once added and detached from the caller's input; providing an > > > existing layer is rejected loudly, add is once-only. Index references > > > (dependencies point at token positions) therefore stay valid for the > > > document's lifetime. > > > > > > 4. Gold versus predicted. Documented as a convention on the same > namespace > > > scheme (gold:opennlp:tokens beside opennlp:tokens); with the once-only > > > rule, competing versions always live under distinct keys and never > replace > > > each other. > > > > > > Acceptance criterion adopted, and it is already an observation rather > than > > > a promise: the feature branches on the PoC fork add all of the layers > above > > > without a single edit to the container package. > > > > > > Given this feedback, mind if I fold points 1 through 4 into the spec > text? > > > Not finalizing, just updating. > > > > > > Rungs, > > > Kristian > > > > > > On Thu, Jul 16, 2026 at 1:22 PM Richard Zowalla <[email protected]> > wrote: > > > > > >> Thanks for the summary. I did some homework on how other systems solve > > >> this before forming an opinion. Short version: the proposed design is > not > > >> novel architecture. That is its strength. The same pattern has emerged > > >> independently in at least four mature systems. > > >> > > >> INCEpTION defines no container of its own. It sits on the UIMA CAS > > >> [1][2]. The CAS holds the original text and typed feature structures > with > > >> begin and end offsets into it. That is classic standoff annotation, > the > > >> same rule the proposal states for spans. What INCEpTION calls a layer > is a > > >> UIMA type plus configurable behaviors. It distinguishes span layers, > > >> relation layers, chain layers for coreference, and metadata layers > without > > >> offsets. The difference is weight. UIMA declares its type system in > XML > > >> descriptors and carries reflection and serialization machinery with > it. The > > >> proposal is deliberately the lightweight end of that spectrum. For a > > >> library, as opposed to an annotation platform, that is the right > decision, > > >> IMHO. > > >> > > >> Stanford CoreNLP is the closest match (to us), almost line for line. > Its > > >> Annotation is a type safe heterogeneous map in the sense of Bloch > > >> [3][4][5]. Keys are class objects parameterized by the value type. The > > >> container never enumerates keys. Any module adds a new key without > touching > > >> the core. The proposed LayerKey is the same pattern with string > identity > > >> instead of class identity. That design has held up in Standford's > CoreNLP > > >> for over fifteen years. > > >> > > >> spaCy has one owning Doc object, spans as views into it, and > doc.spans as > > >> a dict of named span groups under open string keys [6]. Pipeline > components > > >> declare assigns and requires metadata, and the pipeline analyzer > validates > > >> ordering. That is a direct precedent for the proposed requires and > provides > > >> methods. GATE uses named annotation sets with untyped feature maps > [7]. It > > >> is open ended like the proposal but stringly typed in its values, > which the > > >> Class parameter in LayerKey correctly avoids. > > >> > > >> So the field has converged four times on the same shape: standoff > spans > > >> over immutable original text, open keyed typed layers, a container > ignorant > > >> of payloads, and pipelines ordered by declared dependencies. > > >> The guardrail note in the ticket, that the doc container must never > learn > > >> about specific layers, is the single most important sentence in it. > > >> > > >> Four points where the precedents diverge and where I would tighten the > > >> spec before the freeze: > > >> > > >> 1. Key identity. CoreNLP gets collision free identity from class > objects. > > >> String ids do not. Two extensions can both mint "entities". The spec > should > > >> define key equality and adopt namespaced ids for the standard keys, > IMHO. > > >> 2. Annotations without spans. Language id, document categories, > > >> provenance. Every precedent grew a home for these. UIMA has types > without > > >> offsets, spaCy has doc level attributes, INCEpTION has metadata > layers. > > >> Retrofitting this later creates exactly the parallel surface the > proposal > > >> wants to prevent, so we should think about that upfront. > > >> 3. Index based references need stated invariants. UIMA and CoreNLP > > >> reference annotations by object identity. Indices are better for > > >> immutability and serialization, but only if layer order is defined and > > >> existing layers are never mutated or reordered. The spec should - > > >> therefore- also say what happens when an annotator provides a layer > that > > >> already exists. This needs to be defined. > > >> 4. Gold versus predicted layers: GATE annotation sets and UIMA views > > >> exist because corpora might carry competing versions of the same > layer. > > >> Open string keys already handle this. The convention should simply be > > >> documented. > > >> > > >> At a first glance, the validatio plan from the Jira looks right. > > >> Dependency parsing is precisely the consumer that stresses cross layer > > >> references, and coreference will inherit the same rules. > > >> I would also make the acceptance criterion, that a new layer can be > added > > >> without touching the container package. > > >> > > >> Gruß > > >> Richard > > >> > > >> [1] INCEpTION Developer Guide. > > >> > https://inception-project.github.io/releases/26.8/docs/developer-guide.html > > >> [2] Klie JC, Bugert M, Boullosa B, Eckart de Castilho R, Gurevych I. > The > > >> INCEpTION Platform: Machine Assisted and Knowledge Oriented > Interactive > > >> Annotation. COLING 2018. https://aclanthology.org/C18-2002.pdf > > >> [3] TypesafeMap. Stanford CoreNLP API. > > >> > https://nlp.stanford.edu/nlp/javadoc/javanlp-3.5.0/edu/stanford/nlp/util/TypesafeMap.html > > >> [4] Annotation. Stanford CoreNLP API. > > >> > https://nlp.stanford.edu/nlp/javadoc/javanlp-3.5.0/edu/stanford/nlp/pipeline/Annotation.html > > >> [5] CoreAnnotation. Stanford CoreNLP API. > > >> > https://nlp.stanford.edu/nlp/javadoc/javanlp-3.5.0/edu/stanford/nlp/ling/CoreAnnotation.html > > >> [6] spaCy Doc API. https://spacy.io/api/doc > > >> [7] GATE Documentation, Language Resources: Corpora, Documents and > > >> Annotations. https://gate.ac.uk/sale/tao/splitch5.html > > >> > > >> > Am 15.07.2026 um 21:33 schrieb Kristian Rickert <[email protected] > >: > > >> > > > >> > I accidentally sent a draft. To clear things up, I’ve opened > > >> OPENNLP-1888 > > >> > to spark a design discussion around introducing a generic Document > type > > >> > into the 3.0 API. > > >> > > > >> > Currently, our Java API is tool-by-tool; as we look toward adding > more > > >> > complex, structured results (like dependency arcs or relations), > > >> continuing > > >> > this pattern means we'll end up building parallel surface areas for > > >> every > > >> > new capability. > > >> > > > >> > The proposal in the JIRA ticket aims to solve this by introducing a > > >> small, > > >> > unified container in opennlp-api that uses typed annotation layers > over > > >> the > > >> > original text. I want to emphasize right away that this deprecates > and > > >> > replaces absolutely nothing in the existing task-level API; it > simply > > >> > provides a clean, additive way to handle combined-document views > moving > > >> > forward. I'd really love it if we could take a look at the full > > >> > architecture, constraints, and motivations in the ticket and share > your > > >> > thoughts. > > >> > > > >> > I think this design covers what you'd see in other APIs as well as > some > > >> new > > >> > frontier ideas. > > >> > > > >> > Thoughts? > > >> > > > >> > On Wed, Jul 15, 2026 at 3:17 PM Richard Zowalla < > [email protected]> > > >> wrote: > > >> > > > >> >> Did this mail get mangled? > > >> >> > > >> >>> Am 15.07.2026 um 00:17 schrieb Kristian Rickert < > [email protected] > > >> >: > > >> >>> > > >> >>> Two main points: > > >> >>> > > >> >>> - OPENNLP-1888 has not been started; it was created as a design > > >> >>> discussion regarding a generic document shape I'm recommending. > > >> Details > > >> >> in > > >> >>> the ticket > > >> >>> - Thank you everyone for the detailed feedback. I'll ensure every > PR > > >> has > > >> >>> a section in the documentation. > > >> >>> > > >> >>> OPENNLP-1888 was opened to discuss a generic Document type for > the new > > >> >> API > > >> >>> interface. Details are in the ticket, but here is the initial > shape > > >> and I > > >> >>> would like anyone's thoughts on it: > > >> >>> > > >> >>> public interface Document { > > >> >>> CharSequence text(); // the original > text, > > >> >>> never a normalized form > > >> >>> <T> List<Annotation<T>> get(LayerKey<T> layer); // empty list > when the > > >> >>> layer is absent > > >> >>> Set<LayerKey<?>> layers(); > > >> >>> } > > >> >>> > > >> >>> public record Annotation<T>(Span span, T value) { } > > >> >>> > > >> >>> public final class LayerKey<T> { > > >> >>> public static <T> LayerKey<T> of(String id, Class<T> type) { ... } > > >> >>> } > > >> >>> > > >> >>> public interface DocumentAnnotator { > > >> >>> Document annotate(Document document); // returns a new > > >> >> document > > >> >>> with layers added > > >> >>> Set<LayerKey<?>> requires(); > > >> >>> Set<LayerKey<?>> provides(); > > >> >>> } > > >> >> > > >> > > >> > > >
