Re: enhancing module info to allow grouping of firmwares
On Thu, 16 Mar 2023, Dave Airlie wrote: > Hey moduly/firmware people, > > We are facing a problem in the future of NVIDIA providing giant > firmwares for their devices that are version locked with unstable > APIs. Even if they weren't version locked we'd likely have to support > multiple major versions over time. > > Now this becomes a problem because when you generate multiple > initramfs and stick them into /boot over time the number of firmwares > MODULE_FIRMWARE will match will increase and dracut will shove them > all into the initramfs. > > I think one way to mitigate that would be to provide some sort of > grouping of module firmwares that are acceptable, and having dracut > only pick one from the list to provide in the initramfs (hopefully the > latest one). That way the driver can provide a list of MODULE_FIRMWARE > lines and userspace knows they are a group. > > I've just little idea how we could expose this via current module > info, happy to try and come up with an enhanced scheme maybe with a > fallback to just include all of them but was just wanting to get some > feedback on whether this was something that anyone has ever considered > before now. A related problem is platform (or hardware generation) specific firmware blobs that are listed with MODULE_FIRMWARE, and thus added to the generic initramfs. What if there was a way to limit them to the specific platform you have? Sure, a generic initramfs would need them all, but the vast majority of installs would only need the firmware for the hardware on the system. See 'ls /lib/firmware/i915/ | grep -o "^..." | sort | uniq' I don't want to distract you from your original goal, but if you're adding some grouping mechanism, maybe try to keep an avenue open for grouping and filtering like that too? BR, Jani. -- Jani Nikula, Intel Open Source Graphics Center
Re: enhancing module info to allow grouping of firmwares
On Thu, Mar 16, 2023 at 07:18:11AM +1000, Dave Airlie wrote:
> MODULE_FIRMWARE_GROUP("g1")
> MODULE_FIRMWARE("fwv1.bin")
> MODULE_FIRMWARE("fwv2.bin")
> MODULE_FIRMWARE_GROUP_END("g2")
The way module namespaces were implemented could be used to leverage
something like this, just that you'd use it for firmware tags, not
symbols.
drivers/dma-buf/dma-buf.c:EXPORT_SYMBOL_NS_GPL(dma_buf_export, DMA_BUF);
Just that that would look like something like this, which might be
even nicer:
MODULE_FIRMWARE_GROUP("fwv1.bin", "nvidia/g1");
MODULE_FIRMWARE_GROUP("fwv2.bin", "nvidia/g1");
Luis
Re: enhancing module info to allow grouping of firmwares
On Thu, 16 Mar 2023 at 06:56, Alex Deucher wrote:
>
> On Wed, Mar 15, 2023 at 4:35 PM Dave Airlie wrote:
> >
> > Hey moduly/firmware people,
> >
> > We are facing a problem in the future of NVIDIA providing giant
> > firmwares for their devices that are version locked with unstable
> > APIs. Even if they weren't version locked we'd likely have to support
> > multiple major versions over time.
> >
> > Now this becomes a problem because when you generate multiple
> > initramfs and stick them into /boot over time the number of firmwares
> > MODULE_FIRMWARE will match will increase and dracut will shove them
> > all into the initramfs.
> >
> > I think one way to mitigate that would be to provide some sort of
> > grouping of module firmwares that are acceptable, and having dracut
> > only pick one from the list to provide in the initramfs (hopefully the
> > latest one). That way the driver can provide a list of MODULE_FIRMWARE
> > lines and userspace knows they are a group.
> >
> > I've just little idea how we could expose this via current module
> > info, happy to try and come up with an enhanced scheme maybe with a
> > fallback to just include all of them but was just wanting to get some
> > feedback on whether this was something that anyone has ever considered
> > before now.
>
> What about bundling the compatible FWs together into one big FW
> package and tag it with some sort of common metadata header that the
> driver can parse. That would keep all cross FW compatibilities
> together. Then on the driver side, change the firmware name specified
> in the kernel to match whatever is required for that kernel version.
> E.g., one kernel could specify nv_fw_1_0.bin and another could specify
> nv_fw_2_1.bin, etc. It's pretty ugly and not a great user experience
> if there is no backwards compat, but it should work as only the
> compatible FW would be copied to the initrd.
The main thing here is to reduce on-disk size, which I don't think
this will achieve.
I'd rather have the driver say it can support v1, v2, v3 fw and dracut
just put v3 in the initramfs if it's available, else put v2 in etc.
Obviously for newer hw support people will have to ship new fw
versions, and if something gets deprecated then we'd have to move the
fw for that to an always include list.
I envisaged something like
MODULE_FIRMWARE_GROUP("g1")
MODULE_FIRMWARE("fwv1.bin")
MODULE_FIRMWARE("fwv2.bin")
MODULE_FIRMWARE_GROUP_END("g2")
or
MODULE_FIRMWARE("fwv1.bin#g1")
MODULE_FIRMWARE("fwv2.bin#g2")
I also don't want to encourage any firmware teams from abandoning the
stable API principles I think this is mostly to deal with
hostile/semi-hostile and non-vendor drivers where we have no control.
It would be nice though for the couple of amdgpu hiccups we've had over time.
Dave.
>
> Alex
>
>
> >
> > Thanks,
> > Dave.
Re: enhancing module info to allow grouping of firmwares
On Wed, Mar 15, 2023 at 4:35 PM Dave Airlie wrote: > > Hey moduly/firmware people, > > We are facing a problem in the future of NVIDIA providing giant > firmwares for their devices that are version locked with unstable > APIs. Even if they weren't version locked we'd likely have to support > multiple major versions over time. > > Now this becomes a problem because when you generate multiple > initramfs and stick them into /boot over time the number of firmwares > MODULE_FIRMWARE will match will increase and dracut will shove them > all into the initramfs. > > I think one way to mitigate that would be to provide some sort of > grouping of module firmwares that are acceptable, and having dracut > only pick one from the list to provide in the initramfs (hopefully the > latest one). That way the driver can provide a list of MODULE_FIRMWARE > lines and userspace knows they are a group. > > I've just little idea how we could expose this via current module > info, happy to try and come up with an enhanced scheme maybe with a > fallback to just include all of them but was just wanting to get some > feedback on whether this was something that anyone has ever considered > before now. What about bundling the compatible FWs together into one big FW package and tag it with some sort of common metadata header that the driver can parse. That would keep all cross FW compatibilities together. Then on the driver side, change the firmware name specified in the kernel to match whatever is required for that kernel version. E.g., one kernel could specify nv_fw_1_0.bin and another could specify nv_fw_2_1.bin, etc. It's pretty ugly and not a great user experience if there is no backwards compat, but it should work as only the compatible FW would be copied to the initrd. Alex > > Thanks, > Dave.
Re: enhancing module info to allow grouping of firmwares
On Wed, Mar 15, 2023 at 1:35 PM Dave Airlie wrote: > > Hey moduly/firmware people, > > We are facing a problem in the future of NVIDIA providing giant > firmwares for their devices that are version locked with unstable > APIs. Even if they weren't version locked we'd likely have to support > multiple major versions over time. > > Now this becomes a problem because when you generate multiple > initramfs and stick them into /boot over time the number of firmwares > MODULE_FIRMWARE will match will increase and dracut will shove them > all into the initramfs. > > I think one way to mitigate that would be to provide some sort of > grouping of module firmwares that are acceptable, and having dracut > only pick one from the list to provide in the initramfs (hopefully the > latest one). That way the driver can provide a list of MODULE_FIRMWARE > lines and userspace knows they are a group. > > I've just little idea how we could expose this via current module > info, happy to try and come up with an enhanced scheme maybe with a > fallback to just include all of them but was just wanting to get some > feedback on whether this was something that anyone has ever considered > before now. We have a kind of similar problem with drm/msm and various other SoC drivers that need fw which is signed with a device or vendor specific tree. We've partially solved that with using firmware-name from devicetree to specify the correct device specific fw needs to be loaded (ie. something like "qcom/LENOVO/81JL/qcdxkmsuc850.mbn") but I've no idea how dracut should figure out what fw files should end up in an initrd BR, -R
