claudevdm opened a new pull request, #40749:
URL: https://github.com/apache/beam/pull/40749
Adds the two self-contained pieces that dynamic destinations for
`IcebergIO.AddFiles` are
built on: a path-pattern matcher, and the `DoFn` that uses it to name a
destination table
from each file's path.
`AddFiles` registers every input file into one table named by `table`.
Migrating a lake
usually means a tree of per-table folders, which today needs one `AddFiles`
per table.
The goal is for one transform to route each file to the table its path names:
path_pattern: gs://lake/export/{table}/**
table: default.{table}
gs://lake/export/orders/part-0.parquet -> default.orders
gs://lake/export/customers/part-0.parquet -> default.customers
This PR adds only the routing primitives so they can be reviewed on their
own. The
pre-pass, registration and schema-transform surface follow separately.
### `PathPattern`
A whole-path glob with `{name}` placeholders, each capturing exactly one
path segment.
Globbing is Beam's own (`FileSystemUtils.wildcardToRegexp`), so `*`, `**`
and `?` behave as
they do in any other Beam file pattern; placeholders are translated to
capturing groups
between the globbed fixed parts.
The whole path must match, so the pattern decides both *which* table a file
belongs to and
*whether* it belongs at all. `compile` rejects patterns that cannot answer
that
unambiguously:
| rejected | reason |
|---|---|
| `{a}` twice | a name can only be captured once |
| `{a b}`, `{a.b}` | a capture is rendered into a dotted table identifier,
so no dots, slashes or blanks |
| `gs://lake/**/{tbl}/**` | a `**` above a placeholder leaves the captured
folder undetermined |
| `gs://lake/{tbl}/` | ends with `/`, so it names a folder and matches no
file |
`render(template, captures)` fills the table template, and
`checkRenderable(template)`
refuses a template naming a placeholder the pattern never captures, so
`table:
default.{db}` against `.../{tbl}/**` fails at construction rather than
producing one error
row per file.
### `RouteFiles`
`DoFn<String, KV<String, String>>` with two outputs:
- `ROUTED`: `KV<table, path>` for a file the pattern matched
- `ERRORS`: a `{file, error, table}` row for a path it did not, with `table`
empty because
there is no table to attribute it to
An unmatched path becomes an error row rather than being silently dropped: a
mistyped
pattern must be visible. `PathPattern.compile` and `checkRenderable` both
run in the
constructor, so a bad pattern or template fails at pipeline construction
instead of once
per file on a worker.
- `RouteFiles.ERROR_SCHEMA` is declared locally here. When the transform is
wired up it
becomes the same schema as `AddFiles`' error output, so the two can be
flattened; that
change belongs with the wiring.
- Patterns are interpreted by Beam's Java glob translator. Note it
normalises `**/*` to
`**`, which Python's `FileSystem.translate_pattern` does not. Relevant
only if you
compare a `path_pattern` against a glob written for Python's `MatchFiles`
upstream.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
