In the Linux kernel, the following vulnerability has been resolved:
iommu/amd: Fix ineffective error check in nested domain allocation
amd_iommu_pdom_id_alloc() returns an int: a domain ID on success, or the negative errno from ida_alloc_range() when the ID space is exhausted or memory is short. amd_iommu_alloc_domain_nested() stores that return value in gdom_info->hdom_id, which is a u32, and only then tests it:
gdom_info->hdom_id = amd_iommu_pdom_id_alloc(); if (gdom_info->hdom_id <= 0) {
The assignment discards the sign, so -ENOSPC becomes 0xffffffe4 and the test never fires. The nested domain is then set up with a host domain ID that was never allocated, instead of the allocation failing with -ENOSPC.
Keep the value in an int, test it there, and store it only once it is known to be valid, which is what the other amd_iommu_pdom_id_alloc() callers already do.