Andrew Kyle Purtell created HBASE-25824:
-------------------------------------------
Summary: IntegrationTestLoadCommonCrawl
Key: HBASE-25824
URL: https://issues.apache.org/jira/browse/HBASE-25824
Project: HBase
Issue Type: Test
Components: integration tests
Reporter: Andrew Kyle Purtell
Assignee: Andrew Kyle Purtell
This integration test loads successful resource retrieval records from the
Common Crawl (https://commoncrawl.org/) public dataset into an HBase table and
writes records that can be used to later verify the presence and integrity of
those records.
Run like:
{noformat}
./bin/hbase org.apache.hadoop.hbase.test.IntegrationTestLoadCommonCrawl \
-Dfs.s3n.awsAccessKeyId=<AWS access key> \
-Dfs.s3n.awsSecretAccessKey=<AWS secret key> \
/path/to/test-CC-MAIN-2021-10-warc.paths.gz \
/path/to/tmp/warc-loader-output
{noformat}
Access to the Common Crawl dataset in S3 is made available to anyone by Amazon
AWS, but Hadoop's S3N filesystem still requires valid access credentials to
initialize.
The input path can either specify a directory or a file. The file may
optionally be compressed with gzip. If a directory, the loader expects the
directory to contain one or more WARC files from the Common Crawl dataset. If a
file, the loader expects a list of Hadoop S3N URIs which point to S3 locations
for one or more WARC files from the Common Crawl dataset, one URI per line.
Lines should be terminated with the UNIX line terminator.
Included in hbase-it/src/test/resources/CC-MAIN-2021-10-warc.paths.gz is a list
of all WARC files comprising the Q1 2021 crawl archive. There are 64,000 WARC
files in this data set, each containing ~1GB of gzipped data. The WARC files
contain several record types, such as metadata, request, and response, but we
only load the response record types. If the HBase table schema does not specify
compression (by default) there is roughly a 10x expansion. Loading the full
crawl archive results in a table approximately 640 TB in size.
The hadoop-aws jar will be needed at runtime to instantiate the S3N filesystem.
Use the -files ToolRunner argument to add it.
You can also split the Loader and Verify stages:
Load with:
{noformat}
./bin/hbase
'org.apache.hadoop.hbase.test.IntegrationTestLoadCommonCrawl$Loader' \
-files /path/to/hadoop-aws.jar \
-Dfs.s3n.awsAccessKeyId=<AWS access key> \
-Dfs.s3n.awsSecretAccessKey=<AWS secret key> \
/path/to/test-CC-MAIN-2021-10-warc.paths.gz \
/path/to/tmp/warc-loader-output
{noformat}
Verify with:
{noformat}
./bin/hbase
'org.apache.hadoop.hbase.test.IntegrationTestLoadCommonCrawl$Verify' \
/path/to/tmp/warc-loader-output
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
