hanke580 opened a new issue, #10739: URL: https://github.com/apache/paimon/issues/10739
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version master 61cc2dd78443332f8495ec904a87d7b6527aa96c (pypaimon `2.2.dev20261018`, installed with `pip install ./paimon-python`); also pypaimon 2.1.0 from PyPI. ### Compute Engine None: pypaimon (Python API), filesystem catalog with a local `warehouse` path, default options (no native read/write/commit), Python 3.12, Linux 5.15, ext4. ### Minimal reproduce step `commit.py`: ```python import sys import pyarrow as pa from pypaimon import CatalogFactory, Schema catalog = CatalogFactory.create({"warehouse": sys.argv[1]}) schema = pa.schema([("id", pa.int64()), ("v", pa.string())]) catalog.create_database("db", True) catalog.create_table("db.t", Schema.from_pyarrow_schema(schema), True) table = catalog.get_table("db.t") wb = table.new_batch_write_builder() w, c = wb.new_write(), wb.new_commit() w.write_arrow(pa.table({"id": [1, 2, 3], "v": ["a", "b", "c"]}, schema=schema)) c.commit(w.prepare_commit()) w.close(); c.close() print("committed", flush=True) ``` ``` $ strace -f -o trace.txt -e trace=openat,rename,renameat,renameat2,fsync,fdatasync,sync_file_range,syncfs python commit.py /tmp/wh committed $ grep -c -E 'fsync|fdatasync|sync_file_range|syncfs' trace.txt 0 $ grep /tmp/wh trace.txt | grep -E 'O_CREAT|rename' | sed 's/^[0-9]* *//' openat(AT_FDCWD, "/tmp/wh/db.db/t/schema/.schema-0.4f301886-fa09-4185-b0ad-139477065cf1.tmp", O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 3 rename("/tmp/wh/db.db/t/schema/.schema-0.4f301886-fa09-4185-b0ad-139477065cf1.tmp", "/tmp/wh/db.db/t/schema/schema-0") = 0 openat(AT_FDCWD, "/tmp/wh/db.db/t/bucket-0/data-db5a340f-e90e-4e3c-a408-b49b29f4b76b-0.parquet", O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 3 openat(AT_FDCWD, "/tmp/wh/db.db/t/manifest/manifest-3ed1f306-e0b2-44eb-82b7-aaa883014897-0", O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 3 openat(AT_FDCWD, "/tmp/wh/db.db/t/manifest/manifest-list-a30f6c7d-aea7-4c98-84ce-f11371e5d583-1", O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 3 openat(AT_FDCWD, "/tmp/wh/db.db/t/manifest/manifest-list-a30f6c7d-aea7-4c98-84ce-f11371e5d583-0", O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 3 openat(AT_FDCWD, "/tmp/wh/db.db/t/snapshot/.snapshot-1.8235a135-0628-48cf-be00-a2a2bb523a23.tmp", O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 3 rename("/tmp/wh/db.db/t/snapshot/.snapshot-1.8235a135-0628-48cf-be00-a2a2bb523a23.tmp", "/tmp/wh/db.db/t/snapshot/snapshot-1") = 0 openat(AT_FDCWD, "/tmp/wh/db.db/t/snapshot/.LATEST.3a889976-d771-4a67-95fa-d69fb0c82c98.tmp", O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 3 rename("/tmp/wh/db.db/t/snapshot/.LATEST.3a889976-d771-4a67-95fa-d69fb0c82c98.tmp", "/tmp/wh/db.db/t/snapshot/LATEST") = 0 ``` Nothing in a commit (data file, manifests, schema, snapshot, `LATEST`) is synced, and the snapshot is published by renaming a temporary file that was never synced. pypaimon 2.1.0 gives the same trace (0 sync calls). Power-loss test: the warehouse was on ext4 over a device-mapper `linear` device that is switched to the `error` target without a flush after the commits return (everything the kernel had not written to the device is dropped, as in a power cut); the file system was then mounted again and the table read with `get_table("db.t")` and a full scan. The operations: create database and table, append 300 rows, append 300, append 300, overwrite with 100 rows, append 300. On master: | before the cut | result | |---|---| | create + 0 or 2 appends | `TableNotExistException: Table db.t does not exist` | | create + 2 appends, ext4 journal commit (ext4 commits every 5 s by default) | `RuntimeError: Failed to load schema from path: .../db.db/t/schema` | | create + 2 appends made durable with `sync -f`, then append, overwrite, append, journal commit | `JSONDecodeError: Expecting value: line 1 column 1 (char 0)`; `snapshot-3`, `snapshot-4`, `snapshot-5` are 0 bytes, `snapshot-1`/`-2` are intact | | the same with the first 5 operations durable, only the last append not | `JSONDecodeError` (the one unsynced commit makes the table unreadable) | | any of these with a plain process kill instead of a power cut | all rows read correctly | The last part is easy to see without a power cut by truncating the newest snapshot file to the state the cut leaves. `read.py`: ```python import sys from pypaimon import CatalogFactory t = CatalogFactory.create({"warehouse": sys.argv[1]}).get_table("db.t") rb = t.new_read_builder() print(rb.new_read().to_arrow(rb.new_scan().plan().splits()).num_rows, "rows") ``` ``` $ python commit.py /tmp/wh; python commit.py /tmp/wh; python commit.py /tmp/wh $ python read.py /tmp/wh # full scan of db.t 9 rows $ truncate -s0 /tmp/wh/db.db/t/snapshot/snapshot-3 $ python read.py /tmp/wh ... File ".../pypaimon/snapshot/snapshot_manager.py", line 150, in _get_latest_snapshot_from_filesystem return JSON.from_json(self.file_io.read_file_utf8(snapshot_file), Snapshot) ... json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) ``` ### What doesn't meet your expectations? A commit that has returned should survive a power loss (or at least the table should stay readable at the last durable snapshot). Instead a single unsynced commit makes the whole table, including its older durable snapshots, unreadable until someone deletes the empty files by hand; if the power is lost shortly after `create_table`, the table is gone. ### Anything else? `LocalFileIO` ([local_file_io.py](https://github.com/apache/paimon/blob/61cc2dd78443332f8495ec904a87d7b6527aa96c/paimon-python/pypaimon/filesystem/local_file_io.py)) has no `fsync` anywhere: - `new_output_stream()` returns a plain `open(file_path, 'wb')` ([L127-L137](https://github.com/apache/paimon/blob/61cc2dd78443332f8495ec904a87d7b6527aa96c/paimon-python/pypaimon/filesystem/local_file_io.py#L127-L137)), so data files and manifests are not synced on close. - `try_to_write_atomic()` (used for schema and snapshot files and the `LATEST`/`EARLIEST` hints) writes the temporary file and renames it into place without syncing the file before the rename or the directory after it ([L274-L292](https://github.com/apache/paimon/blob/61cc2dd78443332f8495ec904a87d7b6527aa96c/paimon-python/pypaimon/filesystem/local_file_io.py#L274-L292)). On ext4 (and other delayed-allocation file systems) the rename reaches the journal within seconds, while the file's data is written back only later (about 30 s by default), so after a crash the new name points to an empty file. `_get_latest_snapshot_from_filesystem()` then picks the highest snapshot id and parses it without a fallback ([snapshot_manager.py#L95-L150](https://github.com/apache/paimon/blob/61cc2dd78443332f8495ec904a87d7b6527aa96c/paimon-python/pypaimon/snapshot/snapshot_manager.py#L95-L150)). Suggested fix: in `LocalFileIO`, `flush()` + `os.fsync()` the file in the output stream's `close()` and in `try_to_write_atomic()` before the rename, and fsync the parent directory after the rename (and after creating new directories). ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
