This is an automated email from the ASF dual-hosted git repository. zabetak pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/hive-test-datasets.git
commit 1bc1f2838d2f0227d9fcb9bf888417bdec1fb0d1 Author: Stamatis Zampetakis <[email protected]> AuthorDate: Wed Sep 9 11:58:35 2026 +0200 Add information about the Iceberg metadata fetched from S3 --- README.md | 41 +++++++++++++++++++++++++++++++++++++++-- 1 file changed, 39 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index e204b06..2fab882 100644 --- a/README.md +++ b/README.md @@ -1,11 +1,12 @@ # Test datasets for Apache Hive A repository holding datasets and dumps that are used for testing Apache Hive. +The data can be found in the GitHub release assets. +They are not under version control since they are in general large and not expected to change very often. +In addition, some parts are in binary format so they cannot be easily inspected by humans. ## Metastore database dumps A collection of database dumps obtained from various versions of Hive. -The dumps can be found in the GitHub release asserts. They are not under version control since they are in general large and not expected to change very often. -In addition, the dumps are in binary format so they cannot be easily inspected by humans. #### Postgres @@ -21,3 +22,39 @@ Nevertheless, the dumps may still contain additional databases and tables but th Metastore dumps from various [TPC-DS](http://www.tpc.org/tpcds/) scale factors. * [metastore_tpcds10tb_3_1_3000](https://github.com/zabetak/hive-test-datasets/releases/download/1.0/metastore_tpcds10tb_3_1_3000.dump.gz) * [metastore_tpcds30tb_3_1_3000](https://github.com/zabetak/hive-test-datasets/releases/download/1.0/metastore_tpcds30tb_3_1_3000.dump.gz) + +## Iceberg + +### Metadata + +A collection of Iceberg table directories holding only metadata and statistics (no data files) meant to be used mainly in planner tests. + +#### TPC-DS + +Metadata dumps from various [TPC-DS](http://www.tpc.org/tpcds/) scale factors. + +##### 10TB metadata from S3 + +Metadata and stats obtained by downloading the content of the following S3 bucket (excluding data files and unrelated content): +``` +export AWS_ACCESS_KEY_ID=**************** +export AWS_SECRET_ACCESS_KEY=************ + +aws s3 sync s3://dw-team-bucket/data/warehouse/tablespace/external/hive/tpcds_partitioned_iceberg_parquet_10000.db/ iceberg_s3_tpcds10tb --exclude "*" --include "*metadata*" +cd iceberg_s3_tpcds10tb +zip -r ../iceberg_s3_tpcds10tb.zip . +``` + +The DDL statements for querying the tables through Hive can be obtained by running the following script inside the downloaded directory. + +```bash +for f in `ls`; do \ + metafile=`ls $f/metadata/*metadata.json | sort | tail -1 | sed 's|.*metadata/||'`; \ + echo "CREATE EXTERNAL TABLE $f STORED BY 'org.apache.iceberg.mr.hive.HiveIcebergStorageHandler' TBLPROPERTIES('metadata_location'='s3://dw-team-bucket/data/warehouse/tablespace/external/hive/tpcds_partitioned_iceberg_parquet_10000.db/$f/metadata/$metafile');"; \ +done +``` + +WARNING: The script arbitrarily picks the *current* metadata file based on alphanumeric ordering. Ideally, this information should be obtained from the respective catalog. + +Download: [iceberg_s3_tpcds10tb](https://github.com/zabetak/hive-test-datasets/releases/download/1.1/iceberg_s3_tpcds10tb.zip) +
