I forgot to mention the basics; Apache Nutch is a production-grade, extensible, scalable web crawler built on Apache Hadoop. - https://github.com/apache/nutch - https://nutch.apache.org/
On 2026/09/22 00:26:44 lewis john mcgibbney wrote: > Hi user@, > > I have been working on first-class Apache Nutch support in Atlas so that a > crawl can show up as catalog and lineage, without turning every URL into an > Atlas entity. > > JIRA: https://issues.apache.org/jira/browse/ATLAS-5399 > PR: https://github.com/apache/atlas/pull/748 > > The problem: Nutch crawls can be large. If Atlas stored one entity per URL, > a modest crawl would swamp the catalog. Operators still need to answer > basic questions: which crawl produced this index, which hosts were fetched, > which segment was generated, and how did indexing relate to the crawl? > > What Atlas records: The new Nutch model keeps a bounded set of types: > > - nutch_crawl - the CrawlDb (crawl id + cluster) > - nutch_seedlist - the inject seed directory > - nutch_segment - each segment directory > - nutch_domain - registrable domain (eTLD+1), e.g. apache.org > - nutch_host - hostname, unique per Atlas cluster > - nutch_index_process - one Atlas Process per Nutch IndexingJob > - a generic DataSet - the search collection / index you wrote to > > > Fetch counts live on the crawl-to-host relationship (nutch_crawl_hosts), > not as a forest of page entities. There is no nutch_url type. > > How metadata gets into Atlas: There are two writers on purpose: > > 1. Atlas import bridge (the above ticket). hook-bin/import-nutch.sh > reads CrawlDb (the primary Nutch database), optional HostDB, seeds, and > segments. Hosts come from HostDB when you have run UpdateHostDb (a Nutch > tool); otherwise they are rolled up from CrawlDb. > 2. Nutch Atlas IndexWriter (NUTCH-3210). After you index, Nutch emits > one nutch_index_process whose inputs are the crawl/segment and whose output > is the index DataSet. Lineage in the Atlas UI is then crawl + segment -> > process -> index. > > > Both sides must use the same atlas.cluster.name and crawlId. > > A short walkthrough > > After a crawl (HostDB update recommended): > > $ATLAS_HOME/hook-bin/import-nutch.sh -c <crawlId> -d <crawldb> [-H > <hostdb>] [-s <seedDir>] [-g <segmentsDir>] > > Then run your usual Nutch index job with indexer-atlas enabled. In the > Atlas UI you should see Nutch types in the entity tree and a lineage graph > on the indexing process. > > If you crawl with Nutch and already run Atlas, review comments on the JIRA > or PR are welcome. The companion Nutch change is > https://issues.apache.org/jira/browse/NUTCH-3210 > I'm polishing off that PR soon. > > Thanks, > Lewis > http://people.apache.org/keys/committer/lewismc >
