shuan1026 opened a new pull request, #11986:
URL: https://github.com/apache/ozone/pull/11986

   ## What changes were proposed in this pull request?
   
   Deleting an hsync'ed file that is still open does not remove the writer's 
open key. Since HDDS-10650 the delete marks it with `DELETED_HSYNC_KEY`, so 
that the writer's later `CommitKey` is rejected with `KEY_NOT_FOUND` and 
`OpenKeyCleanupService` removes the key once it expires.
   
   For a single key delete in an FSO bucket the marker was written to two 
different keys. `OMKeyDeleteRequestWithFSO` flags the table cache entry under 
the `openFileTable` key, `getOpenFileName(volumeId, bucketId, parentId, 
fileName, clientId)`. `OMKeyDeleteResponseWithFSO.addToDBBatch` persisted the 
flagged entry under `getOpenKey(volume, bucket, key, clientId)`, the 
`openKeyTable` key format. While the cache entry is alive everything works. 
After the double buffer flushes and the cache epoch is evicted, the real open 
row in RocksDB is unmarked and a stray `/volume/bucket/key/clientId` row sits 
in `openFileTable`. Two things follow:
   
   * The writer's `CommitKey` or next hsync is accepted. The deleted file is 
back in `fileTable` while its blocks are already queued for deletion.
   * If the writer never closes, `OmMetadataManagerImpl.getExpiredOpenKeys` 
reaches the unmarked row after the lease hard limit, finds no `fileTable` entry 
and dereferences null. `OpenKeyCleanupService` catches only `IOException`, so 
every FSO run dies at that row and submits nothing. Expired FSO open keys are 
no longer cleaned up or auto-committed, and a restart does not help.
   
   This patch:
   
   * builds the open key in `OMKeyDeleteResponseWithFSO.addToDBBatch` with 
`getOpenFileName(...)`, from the same volume id, bucket object id, parent 
object id and file name the response already uses for the `fileTable` key, so 
the marker lands on the row the request flagged;
   * makes `getExpiredOpenKeys` treat an expired hsync'ed open key whose 
`fileTable` entry is missing as an orphan: it logs a warning and adds the key 
to the delete list instead of dereferencing it. This also heals clusters that 
already hold unmarked rows. The stray `openKeyTable`-format rows carry the 
marker and are removed by the existing marker branch once they expire.
   
   ## What is the link to the Apache JIRA
   
   https://issues.apache.org/jira/browse/HDDS-16768
   
   ## How was this patch tested?
   
   - New `TestOMKeyDeleteResponseWithFSO#testAddToDBBatchMarksHsyncOpenKey`: 
runs `addToDBBatch` with an hsync'ed open key and checks that the 
`getOpenFileName(...)` row carries `DELETED_HSYNC_KEY` and no `getOpenKey(...)` 
row is written. It fails with `expected: <true> but was: <null>` before the 
response change and passes after it.
   - New `TestOpenKeyCleanupService#testCleanupDeletedHsyncKey`: open, hsync 
and delete a key through the OM write path, wait for the double buffer flush, 
then commit. It fails with `Expected OMException to be thrown, but nothing was 
thrown` before the response change; after it the commit fails with 
`KEY_NOT_FOUND` and the cleanup service removes the open key.
   - New `TestOpenKeyCleanupService#testCleanupHsyncKeyWithoutCommittedFile`: 
plants the state a pre-fix OM left behind (file row gone, unmarked open key, 
marked stray row) and checks that `getExpiredOpenKeys` lists both rows for 
deletion, auto-commits nothing, and the service removes both. It fails with the 
`NullPointerException` in `getExpiredOpenKeys` before the scan change and 
passes after it.
   - CI: https://github.com/shuan1026/ozone/actions/runs/42570010169


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to