On Mon, 1 Jun 2026, Leonid Ravich wrote:
> When the underlying skcipher driver advertises support for multiple
> data units in a single request (CRYPTO_ALG_SKCIPHER_MULTI_DATA_UNIT),
> configure the cipher with cc->sector_size as data_unit_size and
> submit one request per bio instead of one request per sector. This
> removes per-sector overhead in the crypto API hot path: request
> allocation, callback dispatch, completion handling, and SG setup.
>
> The optimisation is enabled automatically at table load when all
> of the following hold:
>
> - the cipher is non-aead (i.e. skcipher);
> - tfms_count is 1 (interleaved per-sector keys would break batching);
> - the IV mode is plain or plain64 (the only modes whose generator
> produces a sequential 64-bit little-endian counter that the cipher
> can extend by adding the data-unit index, matching the convention
> documented in crypto_skcipher_set_data_unit_size());
> - the iv_gen_ops->post() hook is unset (lmk and tcw use it; both are
> already excluded by the IV-mode test, but the explicit check makes
> the assumption durable against future IV modes);
> - dm-integrity is not stacked (no integrity tag or integrity IV);
> - the cipher driver advertises multi-data-unit support.
>
> A new CRYPT_MULTI_DATA_UNIT cipher_flag, set once at construction
> time, gates the multi-data-unit path. The existing per-sector path
> in crypt_convert_block_skcipher() is unchanged; the new
> crypt_convert_block_skcipher_multi() is reached from a small dispatch
> in crypt_convert() and shares the same backlog/-EBUSY/-EINPROGRESS
> flow control with the per-sector path.
>
> Heap-allocated scatterlists are stashed in dm_crypt_request and freed
> in crypt_free_req_skcipher() to avoid races between the synchronous-
> success free path and async-completion reuse from the request pool.
> On -ENOMEM during scatterlist allocation, the bio is requeued via
> BLK_STS_DEV_RESOURCE rather than failed, matching the behaviour of
> the existing -ENOMEM path for crypto request allocation.
>
> Verified end-to-end with a byte-equivalence test: encrypted output of
> plain64 dm-crypt with the multi-data-unit path matches output of the
> single-data-unit path bit-for-bit over a 256 MB device.
>
> Signed-off-by: Leonid Ravich <[email protected]>
Reviewed-by: Mikulas Patocka <[email protected]>
