Lewis John McGibbney created ATLAS-5399:
-------------------------------------------
Summary: Add Apache Nutch metadata model and CrawlDb import bridge
Key: ATLAS-5399
URL: https://issues.apache.org/jira/browse/ATLAS-5399
Project: Atlas
Issue Type: New Feature
Components: atlas-intg
Affects Versions: 3.0.0
Reporter: Lewis John McGibbney
Add first-class [Apache Nutch|http://nutch.apache.org] support in Atlas: a
bootstrap type model and a batch bridge that imports crawl catalog metadata
from an existing Nutch CrawlDb.
*Motivation*
Atlas has no crawler/Nutch types. Operators want to govern crawls, seed lists,
hosts, and registrable domains, and to see lineage from a Nutch crawl/segment
into a search index. URL-level entities are intentionally out of scope: CrawlDb
is URL-keyed at web scale and would not fit Atlas's table-shaped models.
*Scope*
# Bootstrap typedefs under addons/models/7000-Nutch/ (serviceType nutch):
## nutch_crawl, nutch_segment extend DataSet
## nutch_seedlist, nutch_domain, nutch_host extend Asset
## nutch_index_process extends Process (instances created by Nutch, not this
bridge)
## Index sink is a generic DataSet (no Solr/ES typedef set)
## COMPOSITION: crawl to seedlist (1:1), crawl→segments
## AGGREGATION: domain to hosts
## ASSOCIATION: many to many relationships between crawls and hosts with
relationship attributes (fetchedCount, unfetchedCount, indexedCount,
lastFetchTime, lastIndexedTime, maxScore)
## qualifiedName uses name@clusterName
## nutch_domain is eTLD+1 / assigned domain (URLUtil.getDomainName /
crawler-commons EffectiveTldFinder), NOT TLD
## nutch_host is global per cluster (hostname@clusterName)
# addons/nutch-bridge: CLI importer using AtlasClientV2 (username/password or
token). Reads \{crawlId}/crawldb, seed dir, segments listing. Creates crawl,
seedlist, segments, domains, and hosts that have at least one FETCHED URL. Does
not create nutch_index_process.
# Distro packaging, docs (HookNutch), and tests with a small CrawlDb fixture
or mocked CrawlDatum rollup.
*Out of scope*
* URL/page entity types or URL sample payloads
* Nutch REST admin / JobManager types (removed in NUTCH-3165)
* AtlasHook / ATLAS_HOOK / consuming indexer-kafka JSON
* Inject/fetch/parse Process types (catalog relationships only in this first
pass) Creating host/domain/crawl entities from the Nutch IndexWriter (see NUTCH)
See also: NUTCH-3210 (Atlas IndexWriter)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
