krickert commented on PR #1154: URL: https://github.com/apache/opennlp/pull/1154#issuecomment-4940277921
Thanks, the data finding especially. Addressed in 224cfe6d: - Mojibake in the bundled table: fixed. The table is now produced by a committed script (opennlp-geo/dev/derive-populated-places.py) that detects the upstream damage by character-class artifacts and by cross-checking against accent-bearing sibling values, repairs values it can verify (67 containment elements, 5 alternate names, 1 canonical name, each recorded in audited tables in the script), drops one-letter corruptions of a name at an accented position (21 values like Zarich for Zurich), and keeps omitting the question-mark-damaged containment (38, now counted exactly in the header). The canonical name derives from the display name folded to ASCII with the upstream transliteration as fallback and alternate, so AmundseniScott, EdDamer, and Sdid Bouzid are gone and the correct spellings resolve. The script hard-fails on unrecognized anomalies, so a data refresh forces a reviewed decision. 181 of 7342 rows changed; the audit scans every name, alternate, and containment value for ar tifact patterns and asserts the exact header row count. - entitylinker overlap: the Gazetteer and Geocoder javadoc now cross-reference opennlp.tools.entitylinker with the guidance that new location consumers target this seam while EntityLinker remains the generic enrichment contract. entitylinker itself is untouched. - I/O failure channel: lookup, byId, byRegion, and Geocoder.resolve declare throws IOException, matching the EntityLinker precedent; the in-memory implementations document that they never throw it. - String vocabularies: promoted to public constants (GazetteerEntry.FEATURE_CLASS_* and ATTRIBUTE_KEY_*, AttributeValue.SOURCE_UNSPECIFIED). CandidateRanking matches on the constants, and a use-case test shows a third-party gazetteer using them getting the feature-class prior. - Emoji annotation references: removed from published javadoc; the remaining text states only the ISO 3166 fact, naming no unshipped class. - Jar LICENSE wiring: the Natural Earth public-domain section is in the binary distribution LICENSE. On per-jar META-INF/LICENSE, the project convention for bundled data is root plus distribution LICENSE, the same convention the runtime jar's Unicode data files follow, so opennlp-geo stays consistent rather than special-casing one jar. - confidence() overflow: computed in double with clamping to [0, 1]; adversarial test with populations at Long.MAX_VALUE. - population zero sentinel: kept deliberately and documented as such; it mirrors the upstream conventions and the confidence prior documents that it treats 0 as absent evidence. - Singleton: public BundledGazetteer.fromEntries(List) added with the same indexing and ranking; getInstance() stays for the bundled table, and a use-case test builds a custom gazetteer. - indexName silent skip: fails loud at load naming the record; the audit additionally asserts every bundled name folds to a non-empty key. - resolve() re-sort: single-candidate lookups skip the copy and sort; multi-candidate lists are still re-sorted because the seam only promises best-effort ordering from arbitrary gazetteers, with a comment saying so. - Parser fail-loud tests: duplicate attribute keys, leading, trailing, and doubled pipes, and out-of-range longitude. - Row-count floor: the audit parses the count from the header and asserts it exactly. - ASF header on the data file: dropped in favor of the derivation record plus the LICENSE pointer; the file rides rat-excludes like the bundled Unicode data files. - Generation tooling: committed, with the extraction date and upstream commit recorded in the derivation record. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
