hsiang-c opened a new pull request, #9335:
URL: https://github.com/apache/iceberg/pull/9335
### Note to reviewers
- Closes https://github.com/apache/iceberg/issues/8856
- Instead of returning `ManifestFile` in
`BaseAllMetadataTableScan::reachableManifests`, we return a `Pair<Snapshot,
ManifestFile>` from all snapshots.
- `REF_SNAPSHOT_ID` is used by `AllManifestsTable` already, so I chose this
terminology instead of `AS_OF_SNAPSHOT`. I am open to other names as well.
- There are a few unit tests in Spark/Flink where the order of the return
values were not guaranteed. Therefore, I sorted the expected/actual values by
`file_path`.
- In the original PR, Russell commented
> These would allow us to be able to analyze the actual history in
all_entires and all_manifests
I also used `Pair<Snapshot, ManifestFile>` for all_* tables other than
`all_entries` and `all_manifests` but I'm not sure if it is necessary. Please
let me know.
### Sample output
- Insert 4 times, delete all of them, then insert one more time.
- Output is ordered by `reference_snapshot_timestamp_millis`
```
+-------------------+---------------------+-----------------------------------+------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|snapshot_id
|reference_snapshot_id|reference_snapshot_timestamp_millis|status|data_file
|
+-------------------+---------------------+-----------------------------------+------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|4341802652067419299|4341802652067419299 |1725425584897
|1 |{0,
file:/var/folders/vh/xgbp56d10b72cjzlng9h8krc0000gn/T/junit12937496981447592132/data/00000-0-9d6311fb-28da-48e0-b9fa-c8dfd32748b1-0-00001.parquet,
PARQUET, 0, 1, 614, {1 -> 42, 2 -> 43}, {1 -> 1, 2 -> 1}, {1 -> 0, 2 -> 0},
{}, {1 -> [01 00 00 00], 2 -> [61]}, {1 -> [01 00 00 00], 2 -> [61]}, NULL,
[4], NULL, 0}|
|4341802652067419299|971355134943407989 |1725425585080
|1 |{0,
file:/var/folders/vh/xgbp56d10b72cjzlng9h8krc0000gn/T/junit12937496981447592132/data/00000-0-9d6311fb-28da-48e0-b9fa-c8dfd32748b1-0-00001.parquet,
PARQUET, 0, 1, 614, {1 -> 42, 2 -> 43}, {1 -> 1, 2 -> 1}, {1 -> 0, 2 -> 0},
{}, {1 -> [01 00 00 00], 2 -> [61]}, {1 -> [01 00 00 00], 2 -> [61]}, NULL,
[4], NULL, 0}|
|971355134943407989 |971355134943407989 |1725425585080
|1 |{0,
file:/var/folders/vh/xgbp56d10b72cjzlng9h8krc0000gn/T/junit12937496981447592132/data/00000-1-65b2a2db-fc1d-44ee-85d5-2e73a7826d33-0-00001.parquet,
PARQUET, 0, 1, 614, {1 -> 42, 2 -> 43}, {1 -> 1, 2 -> 1}, {1 -> 0, 2 -> 0},
{}, {1 -> [01 00 00 00], 2 -> [62]}, {1 -> [01 00 00 00], 2 -> [62]}, NULL,
[4], NULL, 0}|
|971355134943407989 |8947587060401590385 |1725425585198
|1 |{0,
file:/var/folders/vh/xgbp56d10b72cjzlng9h8krc0000gn/T/junit12937496981447592132/data/00000-1-65b2a2db-fc1d-44ee-85d5-2e73a7826d33-0-00001.parquet,
PARQUET, 0, 1, 614, {1 -> 42, 2 -> 43}, {1 -> 1, 2 -> 1}, {1 -> 0, 2 -> 0},
{}, {1 -> [01 00 00 00], 2 -> [62]}, {1 -> [01 00 00 00], 2 -> [62]}, NULL,
[4], NULL, 0}|
|4341802652067419299|8947587060401590385 |1725425585198
|1 |{0,
file:/var/folders/vh/xgbp56d10b72cjzlng9h8krc0000gn/T/junit12937496981447592132/data/00000-0-9d6311fb-28da-48e0-b9fa-c8dfd32748b1-0-00001.parquet,
PARQUET, 0, 1, 614, {1 -> 42, 2 -> 43}, {1 -> 1, 2 -> 1}, {1 -> 0, 2 -> 0},
{}, {1 -> [01 00 00 00], 2 -> [61]}, {1 -> [01 00 00 00], 2 -> [61]}, NULL,
[4], NULL, 0}|
|8947587060401590385|8947587060401590385 |1725425585198
|1 |{0,
file:/var/folders/vh/xgbp56d10b72cjzlng9h8krc0000gn/T/junit12937496981447592132/data/00000-2-c6c84ab2-06b9-4d49-8239-460e448c3c35-0-00001.parquet,
PARQUET, 0, 1, 614, {1 -> 42, 2 -> 43}, {1 -> 1, 2 -> 1}, {1 -> 0, 2 -> 0},
{}, {1 -> [01 00 00 00], 2 -> [61]}, {1 -> [01 00 00 00], 2 -> [61]}, NULL,
[4], NULL, 0}|
|8947587060401590385|2421236859611202602 |1725425585316
|1 |{0,
file:/var/folders/vh/xgbp56d10b72cjzlng9h8krc0000gn/T/junit12937496981447592132/data/00000-2-c6c84ab2-06b9-4d49-8239-460e448c3c35-0-00001.parquet,
PARQUET, 0, 1, 614, {1 -> 42, 2 -> 43}, {1 -> 1, 2 -> 1}, {1 -> 0, 2 -> 0},
{}, {1 -> [01 00 00 00], 2 -> [61]}, {1 -> [01 00 00 00], 2 -> [61]}, NULL,
[4], NULL, 0}|
|971355134943407989 |2421236859611202602 |1725425585316
|1 |{0,
file:/var/folders/vh/xgbp56d10b72cjzlng9h8krc0000gn/T/junit12937496981447592132/data/00000-1-65b2a2db-fc1d-44ee-85d5-2e73a7826d33-0-00001.parquet,
PARQUET, 0, 1, 614, {1 -> 42, 2 -> 43}, {1 -> 1, 2 -> 1}, {1 -> 0, 2 -> 0},
{}, {1 -> [01 00 00 00], 2 -> [62]}, {1 -> [01 00 00 00], 2 -> [62]}, NULL,
[4], NULL, 0}|
|2421236859611202602|2421236859611202602 |1725425585316
|1 |{0,
file:/var/folders/vh/xgbp56d10b72cjzlng9h8krc0000gn/T/junit12937496981447592132/data/00000-3-d3748dc0-4708-4a16-820d-81e2214765c1-0-00001.parquet,
PARQUET, 0, 1, 614, {1 -> 42, 2 -> 43}, {1 -> 1, 2 -> 1}, {1 -> 0, 2 -> 0},
{}, {1 -> [01 00 00 00], 2 -> [62]}, {1 -> [01 00 00 00], 2 -> [62]}, NULL,
[4], NULL, 0}|
|4341802652067419299|2421236859611202602 |1725425585316
|1 |{0,
file:/var/folders/vh/xgbp56d10b72cjzlng9h8krc0000gn/T/junit12937496981447592132/data/00000-0-9d6311fb-28da-48e0-b9fa-c8dfd32748b1-0-00001.parquet,
PARQUET, 0, 1, 614, {1 -> 42, 2 -> 43}, {1 -> 1, 2 -> 1}, {1 -> 0, 2 -> 0},
{}, {1 -> [01 00 00 00], 2 -> [61]}, {1 -> [01 00 00 00], 2 -> [61]}, NULL,
[4], NULL, 0}|
|4324292129886250863|4324292129886250863 |1725425585433
|2 |{0,
file:/var/folders/vh/xgbp56d10b72cjzlng9h8krc0000gn/T/junit12937496981447592132/data/00000-2-c6c84ab2-06b9-4d49-8239-460e448c3c35-0-00001.parquet,
PARQUET, 0, 1, 614, {1 -> 42, 2 -> 43}, {1 -> 1, 2 -> 1}, {1 -> 0, 2 -> 0},
{}, {1 -> [01 00 00 00], 2 -> [61]}, {1 -> [01 00 00 00], 2 -> [61]}, NULL,
[4], NULL, 0}|
|4324292129886250863|4324292129886250863 |1725425585433
|2 |{0,
file:/var/folders/vh/xgbp56d10b72cjzlng9h8krc0000gn/T/junit12937496981447592132/data/00000-1-65b2a2db-fc1d-44ee-85d5-2e73a7826d33-0-00001.parquet,
PARQUET, 0, 1, 614, {1 -> 42, 2 -> 43}, {1 -> 1, 2 -> 1}, {1 -> 0, 2 -> 0},
{}, {1 -> [01 00 00 00], 2 -> [62]}, {1 -> [01 00 00 00], 2 -> [62]}, NULL,
[4], NULL, 0}|
|4324292129886250863|4324292129886250863 |1725425585433
|2 |{0,
file:/var/folders/vh/xgbp56d10b72cjzlng9h8krc0000gn/T/junit12937496981447592132/data/00000-0-9d6311fb-28da-48e0-b9fa-c8dfd32748b1-0-00001.parquet,
PARQUET, 0, 1, 614, {1 -> 42, 2 -> 43}, {1 -> 1, 2 -> 1}, {1 -> 0, 2 -> 0},
{}, {1 -> [01 00 00 00], 2 -> [61]}, {1 -> [01 00 00 00], 2 -> [61]}, NULL,
[4], NULL, 0}|
|4324292129886250863|4324292129886250863 |1725425585433
|2 |{0,
file:/var/folders/vh/xgbp56d10b72cjzlng9h8krc0000gn/T/junit12937496981447592132/data/00000-3-d3748dc0-4708-4a16-820d-81e2214765c1-0-00001.parquet,
PARQUET, 0, 1, 614, {1 -> 42, 2 -> 43}, {1 -> 1, 2 -> 1}, {1 -> 0, 2 -> 0},
{}, {1 -> [01 00 00 00], 2 -> [62]}, {1 -> [01 00 00 00], 2 -> [62]}, NULL,
[4], NULL, 0}|
|2788755331636060629|2788755331636060629 |1725425585574
|1 |{0,
file:/var/folders/vh/xgbp56d10b72cjzlng9h8krc0000gn/T/junit12937496981447592132/data/00000-4-9ba23c37-daaf-4e93-a9bb-5fd06b954fb2-0-00001.parquet,
PARQUET, 0, 1, 614, {1 -> 42, 2 -> 43}, {1 -> 1, 2 -> 1}, {1 -> 0, 2 -> 0},
{}, {1 -> [01 00 00 00], 2 -> [62]}, {1 -> [01 00 00 00], 2 -> [62]}, NULL,
[4], NULL, 0}|
+-------------------+---------------------+-----------------------------------+------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
