The match set negotiated at attach determines which types an attachment could possibly be served with. Which one should actually be served depends on where the buffer is and other attachments when an importer first pins or maps. So far the first entry was used unconditionally, even if it cannot be served to some importers.
Add dma_buf_ops.select_mapping() under dma_resv from both pin and map paths before the type's map callback to let the exporter choose which to serve based on the current placement and list of attachments. The choice is recorded in attach->selected and does not call select_mapping again across any number of unmaps and remaps, until dma_buf_invalidate_mappings() has been delivered to the attachment. This prevents importer remap from bouncing a buffer between placements - the type cannot change without the exporter explicitly invalidating first. In other words attachments that are never invalidated keep the initially selected type for the life of the attachment. A failed select or map establishes nothing and leaves the choice open. Importers never select. The importer's list order is a preference the exporter may weigh or ignore, and an importer must accept any type it listed. Signed-off-by: Alistair Popple <[email protected]> --- drivers/dma-buf/dma-buf-mapping.c | 66 +++++++++++++++++++++++- drivers/dma-buf/dma-buf.c | 106 ++++++++++++++++++++++++++------------ include/linux/dma-buf-mapping.h | 17 ++++-- include/linux/dma-buf.h | 38 ++++++++++++++ 4 files changed, 190 insertions(+), 37 deletions(-) diff --git a/drivers/dma-buf/dma-buf-mapping.c b/drivers/dma-buf/dma-buf-mapping.c index 4eb0ea930900..ff2dbcca23d9 100644 --- a/drivers/dma-buf/dma-buf-mapping.c +++ b/drivers/dma-buf/dma-buf-mapping.c @@ -271,7 +271,8 @@ EXPORT_SYMBOL_NS_GPL(dma_buf_free_sgt, "DMA_BUF"); * importer entry of the same type that the type's match() callback accepts. * Every such pair is recorded in @args->attach->matches, so the match set * preserves the exporter's priority. The first entry is the default returned - * by dma_buf_attach_map_type(). + * by dma_buf_attach_map_type() until &dma_buf_ops.select_mapping chooses + * otherwise at pin or map time. * * Because the exporter list is walked in order, the exporter controls the * priority of mapping types. @@ -324,6 +325,69 @@ int dma_buf_match_mapping(struct dma_buf_match_args *args, } EXPORT_SYMBOL_NS_GPL(dma_buf_match_mapping, "DMA_BUF"); +/** + * dma_buf_mapping_select - Let the exporter choose the type to map with + * @attach: Attachment about to be pinned or mapped + * + * Called with the dma_resv lock held by dma_buf_pin() and the map functions. + * The first call after attach, and after each dma_buf_invalidate_mappings(), + * calls &dma_buf_ops.select_mapping to pick an entry of the match set given + * the buffer's placement and the other live mappings. Exporters without the + * callback get the first entry. Every other call returns the previous choice + * unchanged, so the mapping type cannot change between unmap and remap without + * the exporter having explicitly invalidated. + * + * Returns: 0 on success or the exporter's error, typically -EBUSY when the + * attachment cannot be served without disturbing a mapping that cannot be + * invalidated. + */ +int dma_buf_mapping_select(struct dma_buf_attachment *attach) +{ + int idx = 0; + + dma_resv_assert_held(attach->dmabuf->resv); + + if (attach->selected) + return 0; + + if (WARN_ON(!attach->nr_matches)) + return -EINVAL; + + if (attach->dmabuf->ops->select_mapping) { + idx = attach->dmabuf->ops->select_mapping(attach); + if (idx < 0) + return idx; + if (WARN_ON(idx >= attach->nr_matches)) + return -EINVAL; + } + + attach->selected_idx = idx; + attach->selected = true; + return 0; +} + +/** + * dma_buf_mapping_unselect - Undo a selection that established nothing + * @attach: Attachment whose pin or map just failed + * @was_selected: attach->selected before the failed pin or map began + * + * A pin or map that fails after dma_buf_mapping_select() may reopen the + * choice only if it was the one that made it. A selection that predates the + * call belongs to a live pin or mapping and must survive, otherwise the next + * pin or map would reselect under it without an invalidate. + */ +void dma_buf_mapping_unselect(struct dma_buf_attachment *attach, + bool was_selected) +{ + dma_resv_assert_held(attach->dmabuf->resv); + + if (was_selected) + return; + + attach->selected = false; + attach->selected_idx = 0; +} + static int dma_buf_sgt_match(struct dma_buf_match_args *args, const struct dma_buf_mapping_match *exp, const struct dma_buf_mapping_match *imp) diff --git a/drivers/dma-buf/dma-buf.c b/drivers/dma-buf/dma-buf.c index b3d1f0f863a5..f052184da5ed 100644 --- a/drivers/dma-buf/dma-buf.c +++ b/drivers/dma-buf/dma-buf.c @@ -1182,14 +1182,22 @@ EXPORT_SYMBOL_NS_GPL(dma_buf_detach, "DMA_BUF"); int dma_buf_pin(struct dma_buf_attachment *attach) { struct dma_buf *dmabuf = attach->dmabuf; + bool selected = attach->selected; int ret = 0; WARN_ON(!attach->importer_ops); dma_resv_assert_held(dmabuf->resv); - if (dmabuf->ops->pin) + ret = dma_buf_mapping_select(attach); + if (ret) + return ret; + + if (dmabuf->ops->pin) { ret = dmabuf->ops->pin(attach); + if (ret) + dma_buf_mapping_unselect(attach, selected); + } return ret; } @@ -1216,42 +1224,19 @@ void dma_buf_unpin(struct dma_buf_attachment *attach) } EXPORT_SYMBOL_NS_GPL(dma_buf_unpin, "DMA_BUF"); -/** - * dma_buf_map_sgt - Returns the scatterlist table of the attachment; - * mapped into _device_ address space. Is a wrapper for map_dma_buf() of the - * dma_buf_ops. - * @attach: [in] attachment whose scatterlist is to be returned - * @direction: [in] direction of DMA transfer - * - * Returns sg_table containing the scatterlist to be returned; returns ERR_PTR - * on error. May return -EINTR if it is interrupted by a signal. - * - * On success, the DMA addresses and lengths in the returned scatterlist are - * PAGE_SIZE aligned. - * - * A mapping must be unmapped by using dma_buf_unmap_sgt(). Note that - * the underlying backing storage is pinned for as long as a mapping exists, - * therefore users/importers should not hold onto a mapping for undue amounts of - * time. - * - * Important: Dynamic importers must wait for the exclusive fence of the struct - * dma_resv attached to the DMA-BUF first. - */ -struct sg_table *dma_buf_map_sgt(struct dma_buf_attachment *attach, - enum dma_data_direction direction) +/* The SGT type must have been selected for the attachment already */ +static struct sg_table * +__dma_buf_map_sgt(struct dma_buf_attachment *attach, + enum dma_data_direction direction) { const struct dma_buf_mapping_sgt_exp_ops *sgt_exp_ops = dma_buf_get_sgt_ops(attach); struct sg_table *sg_table; signed long ret; - might_sleep(); - - if (WARN_ON(!attach || !attach->dmabuf || !sgt_exp_ops)) + if (WARN_ON(!sgt_exp_ops)) return ERR_PTR(-EINVAL); - dma_resv_assert_held(attach->dmabuf->resv); - if (dma_buf_pin_on_map(attach)) { ret = attach->dmabuf->ops->pin(attach); /* @@ -1311,6 +1296,53 @@ struct sg_table *dma_buf_map_sgt(struct dma_buf_attachment *attach, return sg_table; } + +/** + * dma_buf_map_sgt - Returns the scatterlist table of the attachment; + * mapped into _device_ address space. Is a wrapper for map_dma_buf() of the + * dma_buf_ops. + * @attach: [in] attachment whose scatterlist is to be returned + * @direction: [in] direction of DMA transfer + * + * Returns sg_table containing the scatterlist to be returned; returns ERR_PTR + * on error. May return -EINTR if it is interrupted by a signal. + * + * On success, the DMA addresses and lengths in the returned scatterlist are + * PAGE_SIZE aligned. + * + * A mapping must be unmapped by using dma_buf_unmap_sgt(). Note that + * the underlying backing storage is pinned for as long as a mapping exists, + * therefore users/importers should not hold onto a mapping for undue amounts of + * time. + * + * Important: Dynamic importers must wait for the exclusive fence of the struct + * dma_resv attached to the DMA-BUF first. + */ +struct sg_table *dma_buf_map_sgt(struct dma_buf_attachment *attach, + enum dma_data_direction direction) +{ + struct sg_table *sg_table; + bool selected; + int ret; + + might_sleep(); + + if (WARN_ON(!attach || !attach->dmabuf)) + return ERR_PTR(-EINVAL); + + dma_resv_assert_held(attach->dmabuf->resv); + + selected = attach->selected; + ret = dma_buf_mapping_select(attach); + if (ret) + return ERR_PTR(ret); + + sg_table = __dma_buf_map_sgt(attach, direction); + if (IS_ERR(sg_table)) + dma_buf_mapping_unselect(attach, selected); + + return sg_table; +} EXPORT_SYMBOL_NS_GPL(dma_buf_map_sgt, "DMA_BUF"); /** @@ -1455,6 +1487,10 @@ EXPORT_SYMBOL_NS_GPL(dma_buf_attach_revocable, "DMA_BUF"); * * Importers are free to request a new mapping in parallel as this function * returns. + * + * This is also the only point at which an attachment's mapping type + * may change after a pin/map: the core keeps the type chosen by + * &dma_buf_ops.select_mapping until the attachment has been invalidated here. */ void dma_buf_invalidate_mappings(struct dma_buf *dmabuf) { @@ -1462,10 +1498,14 @@ void dma_buf_invalidate_mappings(struct dma_buf *dmabuf) dma_resv_assert_held(dmabuf->resv); - list_for_each_entry(attach, &dmabuf->attachments, node) - if (attach->importer_ops && - attach->importer_ops->invalidate_mappings) - attach->importer_ops->invalidate_mappings(attach); + list_for_each_entry(attach, &dmabuf->attachments, node) { + if (!attach->importer_ops || + !attach->importer_ops->invalidate_mappings) + continue; + + attach->selected = false; + attach->importer_ops->invalidate_mappings(attach); + } } EXPORT_SYMBOL_NS_GPL(dma_buf_invalidate_mappings, "DMA_BUF"); diff --git a/include/linux/dma-buf-mapping.h b/include/linux/dma-buf-mapping.h index 69073d589e54..0ee798a5acdd 100644 --- a/include/linux/dma-buf-mapping.h +++ b/include/linux/dma-buf-mapping.h @@ -118,15 +118,26 @@ int dma_buf_match_mapping(struct dma_buf_match_args *args, * dma_buf_attach_map_type - The match this attachment is mapped with * @attach: the attachment * - * Returns the entry of @attach->matches in force for the attachment: the - * exporter's default, which is the first entry. + * Returns the entry of @attach->matches the attachment is mapped with: the + * one the exporter selected at the last pin or map, or the exporter's + * default, the first entry, before anything was selected. After + * dma_buf_invalidate_mappings() it still names the type a live mapping was + * made with, so a pending unmap can use it, until the next selection. */ static inline const struct dma_buf_mapping_match * dma_buf_attach_map_type(const struct dma_buf_attachment *attach) { - return &attach->matches[0]; + return &attach->matches[attach->selected_idx]; } +/* + * Internal: Selection is made by exporters through &dma_buf_ops.select_mapping + * in dma_buf_pin() or map paths. Importers never call this. + */ +int dma_buf_mapping_select(struct dma_buf_attachment *attach); +void dma_buf_mapping_unselect(struct dma_buf_attachment *attach, + bool was_selected); + /* * DMA Mapped Scatterlist Type * diff --git a/include/linux/dma-buf.h b/include/linux/dma-buf.h index ecfd9b2d2aac..33582713a104 100644 --- a/include/linux/dma-buf.h +++ b/include/linux/dma-buf.h @@ -338,6 +338,32 @@ struct dma_buf_ops { * any time until the final location is resolved during map/pin. */ int (*match_mapping)(struct dma_buf_match_args *args); + + /** + * @select_mapping: + * + * Called with the dma_resv lock held from dma_buf_pin() and from the + * map functions, before the type's map callback, to choose which + * entry of &dma_buf_attachment.matches this attachment is mapped + * with. The exporter settles the buffer's placement here, taking all + * live mappings and pinned attachments into account, and returns the + * entry that placement can serve. + * + * The core records the choice and does not ask again until the exporter + * has delivered &dma_buf_attach_ops.invalidate_mappings to this + * attachment. The exporter must therefore keep the recorded type + * servable: moving the buffer somewhere the selected type cannot be + * produced from requires invalidating first. Attachments that cannot + * be invalidated keep their first choice for life. + * + * Optional. Without it the first entry of the match set, the + * exporter's most preferred, is used. + * + * Returns: the index into &dma_buf_attachment.matches to use, or a + * negative errno. -EBUSY means the attachment cannot be served without + * disturbing a mapping that cannot be invalidated. + */ + int (*select_mapping)(struct dma_buf_attachment *attach); }; /** @@ -562,6 +588,16 @@ struct dma_buf_attach_ops { * entry is the exporter's default; dma_buf_attach_map_type() * returns the entry the attachment is mapped with. * @nr_matches: Number of valid entries in @matches. + * @selected_idx: Index into @matches of the entry the attachment is mapped + * with, chosen by &dma_buf_ops.select_mapping at the first pin + * or map and kept, enforced by the core, until + * &dma_buf_attach_ops.invalidate_mappings is delivered to this + * attachment. Zero, the exporter's default, before the first + * selection. Still valid after an invalidate so that a pending + * unmap sees the type its mapping was made with. + * @selected: True while @selected_idx is in force, i.e. the next pin or map + * must not select again; cleared only by + * dma_buf_invalidate_mappings(). * * This structure holds the attachment information between the dma_buf buffer * and its user device(s). The list contains one attachment struct per device @@ -582,6 +618,8 @@ struct dma_buf_attachment { void *priv; struct dma_buf_mapping_match matches[DMA_BUF_MAX_MATCHES]; unsigned int nr_matches; + unsigned int selected_idx; + bool selected; }; /** -- 2.54.0
