Re: [RFC/PATCH v6 03/12] [alsa-devel] media: Entities, pads and links
Laurent Pinchart wrote: > On Thursday 25 November 2010 10:38:05 Clemens Ladisch wrote: > > MEDIA_ENTITY_TYPE_NODE_ALSA_PCM > > MEDIA_ENTITY_TYPE_NODE_ALSA_MIDI > > MEDIA_ENTITY_TYPE_SUBDEV_ALSA_CONTROL > > I agree about PCM and MIDI, but I'm not sure about controls. If controls are > part of an entity, the entity will be reported through the media controller > API. If information about that entity can be queried through existing APIs > (ALSA, V4L, ...) those APIs should be used. At the moment, ALSA has no API for topology information. > I can certainly imagine a graph with 100 controls, but maybe several > controls can be part of the same node ? There is indeed no strict 1:1 relation; e.g., volume and mute are often part of the same node. So it looks we need some kind of separate ALSA node, which can be associated with mixer controls and/or other information. ALSA already has is a method to query arbitrary (TLV) metadata for mixer controls; the entity/control relationship can be stored in the control. > I think we will need a new ioctl in the media controller API to report > advanced information about an entity. This could be used to report controls > implemented by an ALSA element if ALSA doesn't provide a way to do that > directly. This advanced information would always be specific to the entity type, so maybe this should be part of that subsystem's API. Otherwise, the media_entity would need a callback, or store a pointer to some memory block (which assumes that the information is always constant). > > Furthermore, topology information is also needed for entities not > > associated with a mixer control, such as microphones, speakers, jacks/ > > connectors, and effect units. These entities are defined in the USB and > > HD audio specifications, but are not yet handled by ALSA. > > Agreed, we will need to add new entity types and subtypes for those. The > reason they're not part of this submission is that I wanted real use cases > instead of coming up with a made-up list of entities I think would be useful. The reason that I'm always mentioning the USB and HD audio specs is that those already define entities that should cover practically all of the audio needs. (And, of course, we want to be able to report those entities without having to do too many conversions.) I'll see if I can draw up the ALSA-specific media stuff over the weekend. Regards, Clemens -- To unsubscribe from this list: send the line "unsubscribe linux-media" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC/PATCH v6 03/12] [alsa-devel] media: Entities, pads and links
On Thu, Nov 25, 2010 at 04:29:53PM +0100, Laurent Pinchart wrote: > It depends on how you define nodes. I can certainly imagine a graph with 100 > controls, but maybe several controls can be part of the same node ? On the > video side we've decided to split entities depending on the possible data > paths configurations. As I'm not a fluent ascii-art speaker, please have a > look at pages 4 and 5 of http://www.ideasonboard.org/media/20101103-lpc- > media.pdf Please take a look at: http://www.wolfsonmicro.com/products/WM8994 for an example of a modern smartphone CODEC. This has in the ballpark of 75-100 nodes (I've not counted exactly) represented in the routing graph in the driver today. This does not include any audio routing available in other components such as the CPU. -- To unsubscribe from this list: send the line "unsubscribe linux-media" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC/PATCH v6 03/12] [alsa-devel] media: Entities, pads and links
Hi Mark,
On Thursday 25 November 2010 14:41:35 Mark Brown wrote:
> On Thu, Nov 25, 2010 at 10:38:05AM +0100, Clemens Ladisch wrote:
> > In USB and HD audio devices, all links are immutable, and the routing
> > is controlled by 'selector' entities that activate exactly one of their
> > input pads. In userspace, this entity shows up as a mixer control.
> > I guess it would be possible to map the ACTIVE flag onto these controls.
>
> Ditto for ASoC, mostly.
>
> > Alternatively, entities can have 'mute' mixer controls associated with
> > their pads. In this case, multiple unmuted inputs would be mixed
> > together.
> >
> > ALSA has PCM and MIDI devices, and several types of mixer controls.
> > (It also has hardware dependent and timer devices, but I don't think
> >
> > these would need topology information.) So we need at least these:
> > MEDIA_ENTITY_TYPE_NODE_ALSA_PCM
> > MEDIA_ENTITY_TYPE_NODE_ALSA_MIDI
> > MEDIA_ENTITY_TYPE_SUBDEV_ALSA_CONTROL
> >
> > Furthermore, topology information is also needed for entities not
> > associated with a mixer control, such as microphones, speakers, jacks/
> > connectors, and effect units. These entities are defined in the USB and
> > HD audio specifications, but are not yet handled by ALSA.
>
> All this and more in the embedded case - digital audio link nodes and DSP
> I/O nodes (could possibly do those as digital audio ones) spring to
> mind. Also bear in mind that embedded devices can get *very* large - a
> mobile phone audio system can have of the order of 100 nodes in the
> graph.
It depends on how you define nodes. I can certainly imagine a graph with 100
controls, but maybe several controls can be part of the same node ? On the
video side we've decided to split entities depending on the possible data
paths configurations. As I'm not a fluent ascii-art speaker, please have a
look at pages 4 and 5 of http://www.ideasonboard.org/media/20101103-lpc-
media.pdf
Page 4 shows the internal topology of the OMAP3 ISP. The major blocks in that
diagram are reported as entities. Page 5 shows the internal topology of one of
the blocks, the OMAP3 ISP preview engine. As you can see the pipeline is made
of sub-blocks that implement a single image processing function. As the
pipeline is linear (don't worry about the non-linear part in the beginning,
it's just there to take into account link configurability at the higher level)
we don't export all the sub-blocks as entities, but we expose the controls on
the preview engine entity instead.
> > ALSA devices are not addressed by their device node but with card/device/
> >
> > subdevice numbers; mixer controls have numeric IDs, unique per card:
> > struct {
> >
> > int card;
> > int device;
> > int subdevice;
> >
> > } alsa_device;
> > struct {
> >
> > int card;
> > int numid;
> >
> > } alsa_control;
>
> For the embedded stuff we also have a bunch of stuff in the graph which
> may not be visible to userspace at all at present and would just have a
> string based identifier.
That could be easily added (provided the string is not too long).
--
Regards,
Laurent Pinchart
--
To unsubscribe from this list: send the line "unsubscribe linux-media" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC/PATCH v6 03/12] [alsa-devel] media: Entities, pads and links
On Thu, Nov 25, 2010 at 04:21:38PM +0100, Laurent Pinchart wrote: > On Thursday 25 November 2010 10:38:05 Clemens Ladisch wrote: > > ALSA has PCM and MIDI devices, and several types of mixer controls. > > (It also has hardware dependent and timer devices, but I don't think > > these would need topology information.) So we need at least these: > > MEDIA_ENTITY_TYPE_NODE_ALSA_PCM > > MEDIA_ENTITY_TYPE_NODE_ALSA_MIDI > > MEDIA_ENTITY_TYPE_SUBDEV_ALSA_CONTROL > I agree about PCM and MIDI, but I'm not sure about controls. If controls are > part of an entity, the entity will be reported through the media controller > API. If information about that entity can be queried through existing APIs > (ALSA, V4L, ...) those APIs should be used. For instance, on the V4L side, > V4L2 sub-devices are mapped to entities and also have a device node which can > be used to enumerate the controls supported by the subdev. The media > controller only reports the entity -> major:minor mapping to let applications > find the device and query it directly. For audio we don't currently have a sensible API for associating controls with any sort of map of how the device is laid out, userspace has to play guessing games. > I think we will need a new ioctl in the media controller API to report > advanced information about an entity. This could be used to report controls > implemented by an ALSA element if ALSA doesn't provide a way to do that > directly. Another use case, which make me think that such an ioctl would be > needed, is to report UVC extension units type (16-byte GUID) to userspace. That seems reasonable. -- To unsubscribe from this list: send the line "unsubscribe linux-media" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC/PATCH v6 03/12] [alsa-devel] media: Entities, pads and links
Hi Clemens, Thanks a lot for the review. On Thursday 25 November 2010 10:38:05 Clemens Ladisch wrote: > Laurent Pinchart wrote: > > A link is a point-to-point oriented connection between two pads, either > > on the same entity or on different entities. Data flows from a source > > pad to a sink pad. > > > > Links are stored in the source entity. > > In the descriptors of USB Audio and HDAudio devices, the links are > stored only in the sink. But AFAICS this doesn't matter in the media > driver API. Same for USB Video (UVC). The reason for that is that UAC and UVC have only 1- to-many connections (same for the media controller by the way), and it's easier to store that information in fixed-size structures if you store it in the sink entities. On the kernel side, the media controller stores links in both source and sink entities for internal reasons. Links stored in the sink entity are called backlinks. On the userspace side applications will only see "forward" links, stored in source entities. Applications will of course be free to store the links whereever they want in their internal data structures. As you mentioned, I think it doesn't matter much anyway. > > +Links have flags that describe the link capabilities and state. > > + > > + MEDIA_LINK_FLAG_ACTIVE indicates that the link is active and can be > > + used to transfer media data. When two or more links target a sink pad, > > + only one of them can be active at a time. > > + MEDIA_LINK_FLAG_IMMUTABLE indicates that the link active state can't > > + be modified at runtime. If MEDIA_LINK_FLAG_IMMUTABLE is set, then > > + MEDIA_LINK_FLAG_ACTIVE must also be set since an immutable link is > > + always active. > > So the intended userspace API for controlling routing is to change the > ACTIVE flag of links? That's correct. > In USB and HD audio devices, all links are immutable, and the routing > is controlled by 'selector' entities that activate exactly one of their > input pads. In userspace, this entity shows up as a mixer control. > I guess it would be possible to map the ACTIVE flag onto these controls. Same for UVC once again. It would be quite easy to map selector units to link activation, but it doesn't even have to be done. We could keep the selector units in the graph with all links immutable, and use the existing APIs to control the selector units. The alternative would have been to add selector units to the media controller. I've thought about it, but we have embedded hardware (at least on the video side) that have configurable links without selector units. The media controller would have had to create fake selector units. Between creating fake selector and translating link activation to selector unit commands, I've decided to go for the second solution, as this will make the graph simpler. Once again we can decide to keep explicit selector units in the graph for UVC, UAC and HD audio devices. That's one I've done in a test implementation of the media controller API in the uvcvideo driver. > Alternatively, entities can have 'mute' mixer controls associated with > their pads. In this case, multiple unmuted inputs would be mixed > together. I don't see any problem keeping those. > > +#define MEDIA_ENTITY_TYPE_MASK 0x00ff > > +#define MEDIA_ENTITY_SUBTYPE_MASK0x > > + > > +#define MEDIA_ENTITY_TYPE_NODE (1 << > > MEDIA_ENTITY_TYPE_SHIFT) ... > > +#define MEDIA_ENTITY_TYPE_NODE_ALSA (MEDIA_ENTITY_TYPE_NODE + > > 3) > > ALSA has PCM and MIDI devices, and several types of mixer controls. > (It also has hardware dependent and timer devices, but I don't think > these would need topology information.) So we need at least these: > MEDIA_ENTITY_TYPE_NODE_ALSA_PCM > MEDIA_ENTITY_TYPE_NODE_ALSA_MIDI > MEDIA_ENTITY_TYPE_SUBDEV_ALSA_CONTROL I agree about PCM and MIDI, but I'm not sure about controls. If controls are part of an entity, the entity will be reported through the media controller API. If information about that entity can be queried through existing APIs (ALSA, V4L, ...) those APIs should be used. For instance, on the V4L side, V4L2 sub-devices are mapped to entities and also have a device node which can be used to enumerate the controls supported by the subdev. The media controller only reports the entity -> major:minor mapping to let applications find the device and query it directly. I think we will need a new ioctl in the media controller API to report advanced information about an entity. This could be used to report controls implemented by an ALSA element if ALSA doesn't provide a way to do that directly. Another use case, which make me think that such an ioctl would be needed, is to report UVC extension units type (16-byte GUID) to userspace. > Furthermore, topology information is also needed for entities not > associated with a mixer control, such as microphones, speakers, jacks/ > connectors, and effect units
Re: [RFC/PATCH v6 03/12] [alsa-devel] media: Entities, pads and links
On Thu, Nov 25, 2010 at 10:38:05AM +0100, Clemens Ladisch wrote:
> In USB and HD audio devices, all links are immutable, and the routing
> is controlled by 'selector' entities that activate exactly one of their
> input pads. In userspace, this entity shows up as a mixer control.
> I guess it would be possible to map the ACTIVE flag onto these controls.
Ditto for ASoC, mostly.
> Alternatively, entities can have 'mute' mixer controls associated with
> their pads. In this case, multiple unmuted inputs would be mixed
> together.
> ALSA has PCM and MIDI devices, and several types of mixer controls.
> (It also has hardware dependent and timer devices, but I don't think
> these would need topology information.) So we need at least these:
> MEDIA_ENTITY_TYPE_NODE_ALSA_PCM
> MEDIA_ENTITY_TYPE_NODE_ALSA_MIDI
> MEDIA_ENTITY_TYPE_SUBDEV_ALSA_CONTROL
> Furthermore, topology information is also needed for entities not
> associated with a mixer control, such as microphones, speakers, jacks/
> connectors, and effect units. These entities are defined in the USB and
> HD audio specifications, but are not yet handled by ALSA.
All this and more in the embedded case - digital audio link nodes and DSP
I/O nodes (could possibly do those as digital audio ones) spring to
mind. Also bear in mind that embedded devices can get *very* large - a
mobile phone audio system can have of the order of 100 nodes in the
graph.
> ALSA devices are not addressed by their device node but with card/device/
> subdevice numbers; mixer controls have numeric IDs, unique per card:
>
> struct {
> int card;
> int device;
> int subdevice;
> } alsa_device;
> struct {
> int card;
> int numid;
> } alsa_control;
For the embedded stuff we also have a bunch of stuff in the graph which
may not be visible to userspace at all at present and would just have a
string based identifier.
--
To unsubscribe from this list: send the line "unsubscribe linux-media" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC/PATCH v6 03/12] [alsa-devel] media: Entities, pads and links
Laurent Pinchart wrote:
> A link is a point-to-point oriented connection between two pads, either
> on the same entity or on different entities. Data flows from a source
> pad to a sink pad.
>
> Links are stored in the source entity.
In the descriptors of USB Audio and HDAudio devices, the links are
stored only in the sink. But AFAICS this doesn't matter in the media
driver API.
> +Links have flags that describe the link capabilities and state.
> +
> + MEDIA_LINK_FLAG_ACTIVE indicates that the link is active and can be
> + used to transfer media data. When two or more links target a sink pad,
> + only one of them can be active at a time.
> + MEDIA_LINK_FLAG_IMMUTABLE indicates that the link active state can't
> + be modified at runtime. If MEDIA_LINK_FLAG_IMMUTABLE is set, then
> + MEDIA_LINK_FLAG_ACTIVE must also be set since an immutable link is
> + always active.
So the intended userspace API for controlling routing is to change the
ACTIVE flag of links?
In USB and HD audio devices, all links are immutable, and the routing
is controlled by 'selector' entities that activate exactly one of their
input pads. In userspace, this entity shows up as a mixer control.
I guess it would be possible to map the ACTIVE flag onto these controls.
Alternatively, entities can have 'mute' mixer controls associated with
their pads. In this case, multiple unmuted inputs would be mixed
together.
> +#define MEDIA_ENTITY_TYPE_MASK 0x00ff
> +#define MEDIA_ENTITY_SUBTYPE_MASK0x
> +
> +#define MEDIA_ENTITY_TYPE_NODE (1 << MEDIA_ENTITY_TYPE_SHIFT)
> ...
> +#define MEDIA_ENTITY_TYPE_NODE_ALSA (MEDIA_ENTITY_TYPE_NODE + 3)
ALSA has PCM and MIDI devices, and several types of mixer controls.
(It also has hardware dependent and timer devices, but I don't think
these would need topology information.) So we need at least these:
MEDIA_ENTITY_TYPE_NODE_ALSA_PCM
MEDIA_ENTITY_TYPE_NODE_ALSA_MIDI
MEDIA_ENTITY_TYPE_SUBDEV_ALSA_CONTROL
Furthermore, topology information is also needed for entities not
associated with a mixer control, such as microphones, speakers, jacks/
connectors, and effect units. These entities are defined in the USB and
HD audio specifications, but are not yet handled by ALSA.
> +struct media_entity {
> ...
> + union {
> + /* Node specifications */
> + struct {
> + u32 major;
> + u32 minor;
> + } v4l;
> + struct {
> + u32 major;
> + u32 minor;
> + } fb;
> + int alsa;
> + int dvb;
> +
> + /* Sub-device specifications */
> + /* Nothing needed yet */
> + };
ALSA devices are not addressed by their device node but with card/device/
subdevice numbers; mixer controls have numeric IDs, unique per card:
struct {
int card;
int device;
int subdevice;
} alsa_device;
struct {
int card;
int numid;
} alsa_control;
Regards,
Clemens
--
To unsubscribe from this list: send the line "unsubscribe linux-media" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
