A couple of extra patches on top of Shawn's existing re-ordering patch. This seems to fix the problem Alexey noted with Shawn's change causing VFs to lose their IOMMU group. I've tried pretty hard to make this a minimal fix it's still a bit large.
If mpe is happy to take this as a fix for 5.4 then I'll leave it, otherwise we might want to look at different approaches.