lewismc opened a new pull request, #748: URL: https://github.com/apache/atlas/pull/748
## What changes were proposed in this pull request? Catalog Apache Nutch crawls in Atlas without creating one entity per URL ([ATLAS-5399](https://issues.apache.org/jira/browse/ATLAS-5399)). Adds a Nutch metadata model (`addons/models/7000-Nutch/`) and a CrawlDb/HostDB import bridge (`addons/nutch-bridge`). The bridge creates crawl, seedlist, segment, domain, and host entities, plus per-crawl host metrics on `nutch_crawl_hosts`. Hosts are preferred from Nutch HostDB when `{crawl}/hostdb/current` exists; otherwise they are rolled up from CrawlDb. There is no `nutch_url` type. Indexing lineage is **not** emitted here. The model includes `nutch_index_process`; the Nutch Atlas IndexWriter creates that process and a generic index `DataSet` ([NUTCH-3210](https://issues.apache.org/jira/browse/NUTCH-3210)). The two sides share `atlas.cluster.name` and `crawlId`. Import CLI: ``` hook-bin/import-nutch.sh -c -d [-H ] [-s ] [-g ] ``` The distro `nutch-hook` package ships the bridge and its runtime dependencies so import does not need Nutch on the classpath. Docs: `docs/src/documents/Hook/HookNutch.md`. **Out of scope**: URL/page entities, Nutch REST/JobManager types (NUTCH-3165), `indexer-kafka` / `ATLAS_HOOK`, inject/fetch/parse process types. ## How was this patch tested? - `mvn -pl addons/nutch-bridge test` (CrawlDb MapFile partitions, HostDB SequenceFile, qualified names, host rollup, model JSON) - Live `dev-support/atlas-docker` (postgres) with seed `https://nutch.apache.org/`: import created crawl, seedlist, segment, `apache.org`, `nutch.apache.org`, and crawl-host metrics; Nutch IndexingJob then created the index `DataSet` and `nutch_index_process` - Lineage in the Atlas UI: crawl + segment to `nutch_index_process` to index See attached screenshot of the lineage tab for the demo indexing process. <img width="1677" height="1981" alt="atlas-nutch-lineage" src="https://github.com/user-attachments/assets/e3f4596d-ddc9-4aba-8f4f-f3eee968687e"; /> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
