wombatu-kun commented on code in PR #19282: URL: https://github.com/apache/hudi/pull/19282#discussion_r3628612328
########## website/learn/tech-specs.md: ########## @@ -216,24 +233,55 @@ meaningful for them. ### Log Files -The log files contain different type of blocks, that encode delta updates, deletes or change logs against the base file. They are named with the convention +Log files encode inserts, updates, deletes, or change logs against a file group. Hudi supports two physical log formats, +and a file slice may contain both formats after a table-version upgrade. + +**Inline log file format (v0/v1)** + +Inline logs use the following hidden-file naming convention: + +**_.\[file\_id\]\_\[requested\_instant\_time\].log.\[version\]\_\[write\_token\]_** + +An inline log file is appendable and can contain multiple Hudi-framed blocks. Inline CDC data uses the optional `.cdc` +suffix after the write token. -**_.\[file\_id\]\_\[requested\_instant\_time\].log.\[version\]\_\[write\_token\]_**. +**Native log file format (v2)** +Native logs use a visible, write-once file with this naming convention: +**_\[file\_id\]\_\[write\_token\]\_\[requested\_instant\_time\]\_\[version\].\[log\_type\].\[native\_format\]_** -* **file\_id** - File Id of the base file in the slice -* **requested\_instant\_time** \- Time at which the write operation that produced this log file was requested. -* **version** - Current version of the log file format, to order deltas against the base file. -* **write\_token** - Monotonically increasing token for every attempt to write the log file. This should help uniquely identifying the log file when there are failures and retries. - Cleaner can clean-up partial log files if the write token is not the latest in the file slice. +| Log type | File-name component | Parquet example | +| --- | --- | --- | +| Data | `log` | `<file_id>_<write_token>_<instant>_<version>.log.parquet` | +| Delete | `deletes` | `<file_id>_<write_token>_<instant>_<version>.deletes.parquet` | +| Change data capture | `cdc` | `<file_id>_<write_token>_<instant>_<version>.cdc.parquet` | + +The fields used by both conventions are: + +* **file_id** - ID of the file group to which the log belongs. +* **requested_instant_time** - Begin time of the write action that produced the log. +* **version** - Sequence number that orders log files for the same file group and instant. This is distinct from the + physical log-format version stored in the block envelope or native-file footer. +* **write_token** - Token identifying the task and attempt that produced the file. It distinguishes retries and lets + cleaning remove unsuccessful attempts. +* **native_format** - Storage format of a native log file, such as `parquet`, `orc`, or `hfile`. The current writer uses + the effective base-file format; Lance is the exception described below. + +Table versions below 10 write inline logs. Writers targeting table version 10 or later write native logs by default, Review Comment: The metadata table is not an exception here: it inherits the data table's write version in `HoodieMetadataWriteUtils`, and `HoodieNativeLogAppendHandle` hands the native writer the table's base-file format, so it writes native HFile logs. Point lookups survive that switch through `HoodieNativeAvroHFileReader.supportKeyPredicate`, so no MDT exemption belongs in the list. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
