I have not found a solution to this issue.However there is a workaround that involves passing the serial device to the guest VM instead of the USB device:
https://community.home-assistant.io/t/cannot-passthrough-conbee-ii-with-kvm-host-debian-guest-haos/465496/5 In this specific case I was able to use the zigbee controller with HA.The underlying problem (passthrough not working properly) however still remains.
Greets, Lee