I forgot to mention the basics; Apache Nutch is a production-grade, extensible, 
scalable web crawler built on Apache Hadoop.
- https://github.com/apache/nutch
- https://nutch.apache.org/

On 2026/09/22 00:26:44 lewis john mcgibbney wrote:
> Hi user@,
> 
> I have been working on first-class Apache Nutch support in Atlas so that a
> crawl can show up as catalog and lineage, without turning every URL into an
> Atlas entity.
> 
> JIRA: https://issues.apache.org/jira/browse/ATLAS-5399
> PR: https://github.com/apache/atlas/pull/748
> 
> The problem: Nutch crawls can be large. If Atlas stored one entity per URL,
> a modest crawl would swamp the catalog. Operators still need to answer
> basic questions: which crawl produced this index, which hosts were fetched,
> which segment was generated, and how did indexing relate to the crawl?
> 
> What Atlas records: The new Nutch model keeps a bounded set of types:
> 
>    - nutch_crawl - the CrawlDb (crawl id + cluster)
>    - nutch_seedlist - the inject seed directory
>    - nutch_segment - each segment directory
>    - nutch_domain - registrable domain (eTLD+1), e.g. apache.org
>    - nutch_host - hostname, unique per Atlas cluster
>    - nutch_index_process - one Atlas Process per Nutch IndexingJob
>    - a generic DataSet - the search collection / index you wrote to
> 
> 
> Fetch counts live on the crawl-to-host relationship (nutch_crawl_hosts),
> not as a forest of page entities. There is no nutch_url type.
> 
> How metadata gets into Atlas: There are two writers on purpose:
> 
>    1. Atlas import bridge (the above ticket). hook-bin/import-nutch.sh
>    reads CrawlDb (the primary Nutch database), optional HostDB, seeds, and
>    segments. Hosts come from HostDB when you have run UpdateHostDb (a Nutch
>    tool); otherwise they are rolled up from CrawlDb.
>    2. Nutch Atlas IndexWriter (NUTCH-3210). After you index, Nutch emits
>    one nutch_index_process whose inputs are the crawl/segment and whose output
>    is the index DataSet. Lineage in the Atlas UI is then crawl + segment ->
>    process -> index.
> 
> 
> Both sides must use the same atlas.cluster.name and crawlId.
> 
> A short walkthrough
> 
> After a crawl (HostDB update recommended):
> 
> $ATLAS_HOME/hook-bin/import-nutch.sh -c <crawlId> -d <crawldb> [-H
> <hostdb>] [-s <seedDir>] [-g <segmentsDir>]
> 
> Then run your usual Nutch index job with indexer-atlas enabled. In the
> Atlas UI you should see Nutch types in the entity tree and a lineage graph
> on the indexing process.
> 
> If you crawl with Nutch and already run Atlas, review comments on the JIRA
> or PR are welcome. The companion Nutch change is
> https://issues.apache.org/jira/browse/NUTCH-3210
> I'm polishing off that PR soon.
> 
> Thanks,
> Lewis
> http://people.apache.org/keys/committer/lewismc
> 

Reply via email to