lasdf1234 commented on PR #12241:
URL: https://github.com/apache/gravitino/pull/12241#issuecomment-5178530018
# Manifest entry-range splitting (follow-up)
A `plan-task` is one batch of manifest entry ranges to read.
The opaque string is `base64url(JSON array)`: the root is the **array
itself** — no `ranges` wrapper, and no embedded scan/snapshot.
## `planTaskKey` (decoded)
```json
[
{
"manifest": "s3://wh/db/t/metadata/snap-42-m1.avro",
"entry-start": 40,
"entry-end": 60
},
{
"manifest": "s3://wh/db/t/metadata/snap-42-m2.avro",
"entry-start": 0,
"entry-end": 60
},
{
"manifest": "s3://wh/db/t/metadata/snap-42-m3.avro",
"entry-start": 0,
"entry-end": 20
}
]
```
| Field | Meaning |
| --- | --- |
| `manifest` | Data manifest path |
| `entry-start` / `entry-end` | Half-open range `[start, end)` within that
manifest |
| Array length | `1` = single manifest; `≥ 2` = this batch spans multiple
manifests |
## Setup (example)
```text
batch-size = 100
m0 → 60 entries
m1 → 60 entries
m2 → 60 entries
m3 → 70 entries
```
```text
Batch 0 (100): m0[0,60) + m1[0,40) → inlined on /plan
Batch 1 (100): m1[40,60) + m2[0,60) + m3[0,20) → plan-task
Batch 2 (50): m3[20,70) → plan-task
```
## First `POST .../plan` response
```json
{
"plan-status": "completed",
"file-scan-tasks": [
{ "data-file": { "file-path": "s3://wh/db/t/data/m0-e00.parquet" } },
{ "data-file": { "file-path": "s3://wh/db/t/data/m1-e39.parquet" } }
],
"plan-tasks": [
"W3sibWFuaWZlc3QiOiJzMzovL3doL2RiL3QvbWV0YWRhdGEvc25hcC00Mi1tMS5hdnJvIiwiZW50cnktc3RhcnQiOjQwLCJlbnRyeS1lbmQiOjYwfSx7Im1hbmlmZXN0IjoiczM6Ly93aC9kYi90L21ldGFkYXRhL3NuYXAtNDItbTIuYXZybyIsImVudHJ5LXN0YXJ0IjowLCJlbnRyeS1lbmQiOjYwfSx7Im1hbmlmZXN0IjoiczM6Ly93aC9kYi90L21ldGFkYXRhL3NuYXAtNDItbTMuYXZybyIsImVudHJ5LXN0YXJ0IjowLCJlbnRyeS1lbmQiOjIwfV0=",
"W3sibWFuaWZlc3QiOiJzMzovL3doL2RiL3QvbWV0YWRhdGEvc25hcC00Mi1tMy5hdnJvIiwiZW50cnktc3RhcnQiOjIwLCJlbnRyeS1lbmQiOjcwfV0="
],
"specs-by-id": {
"0": { "spec-id": 0, "fields": [] }
}
}
```
`file-scan-tasks`: Batch 0 — 100 tasks (only head/tail shown).
`plan-tasks[0]` decodes to Batch 1:
```json
[
{ "manifest": "s3://wh/db/t/metadata/snap-42-m1.avro", "entry-start": 40,
"entry-end": 60 },
{ "manifest": "s3://wh/db/t/metadata/snap-42-m2.avro", "entry-start": 0,
"entry-end": 60 },
{ "manifest": "s3://wh/db/t/metadata/snap-42-m3.avro", "entry-start": 0,
"entry-end": 20 }
]
```
`plan-tasks[1]` decodes to Batch 2:
```json
[
{ "manifest": "s3://wh/db/t/metadata/snap-42-m3.avro", "entry-start": 20,
"entry-end": 70 }
]
```
## `POST .../tasks`
### Request
```json
{
"plan-task":
"W3sibWFuaWZlc3QiOiJzMzovL3doL2RiL3QvbWV0YWRhdGEvc25hcC00Mi1tMS5hdnJvIiwiZW50cnktc3RhcnQiOjQwLCJlbnRyeS1lbmQiOjYwfSx7Im1hbmlmZXN0IjoiczM6Ly93aC9kYi90L21ldGFkYXRhL3NuYXAtNDItbTIuYXZybyIsImVudHJ5LXN0YXJ0IjowLCJlbnRyeS1lbmQiOjYwfSx7Im1hbmlmZXN0IjoiczM6Ly93aC9kYi90L21ldGFkYXRhL3NuYXAtNDItbTMuYXZybyIsImVudHJ5LXN0YXJ0IjowLCJlbnRyeS1lbmQiOjIwfV0="
}
```
Server: decode the array → read each slice in order → return that batch’s
`file-scan-tasks` (no full-table re-plan).
### Response
```json
{
"file-scan-tasks": [
{ "data-file": { "file-path": "s3://wh/db/t/data/m1-e40.parquet" } },
{ "data-file": { "file-path": "s3://wh/db/t/data/m2-e00.parquet" } },
{ "data-file": { "file-path": "s3://wh/db/t/data/m3-e00.parquet" } }
],
"plan-tasks": [],
"specs-by-id": {
"0": { "spec-id": 0, "fields": [] }
}
}
```
## Compared with the current PR design
| | PR (offset) | This approach |
| --- | --- | --- |
| Decoded `plan-task` | `{ table, offset, limit, scan }` | JSON **array**
`[{ manifest, entry-start, entry-end }, …]` |
| `/tasks` | May fully re-plan, then slice | Read only those manifest slices
|
| Multi-manifest batch | Crossed via global indexes | Multiple elements in
the array |
## Short PR comment
> Follow-up: let each `plan-task` be `base64url` of a JSON **array** of
`{manifest, entry-start, entry-end}` (no wrapper object, no embedded scan).
`/plan` inlines batch 0; later batches are these opaque arrays. `/tasks` reads
only those manifest slices. A multi-manifest batch is simply a multi-element
array.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
