In the Linux kernel, the following vulnerability has been resolved:
mm/hugetlb: restore failed global reservations to subpool
Commit a833a693a490 ("mm: hugetlb: fix incorrect fallback for subpool") fixed an underflow error for hstate->resv_huge_pages caused by incorrectly attributing globally requested pages to the subpool's reservation.
Unfortunately, this fix also introduced the opposite problem, which would leave spool->used_hpages elevated if the globally requested pages could not be acquired. This is because while a subpool's reserve pages only accounts for what is requested and allocated from the subpool, its "used" counter keeps track of what is consumed in total, both from the subpool and globally. Thus, we need to adjust spool->used_hpages in the other direction, and make sure that globally requested pages are uncharged from the subpool's used counter.
Each failed allocation attempt increments the used_hpages counter by how many pages were requested from the global pool. Ultimately, this renders the subpool unusable, as used_hpages approaches the max limit.
The issue can be reproduced as follows:
- Allocate 4 hugetlb pages
- Create a hugetlb mount with max=4, min=2
- Consume 2 pages globally
- Request 3 pages from the subpool (2 from subpool + 1 from global)
- Subpool now has used_hpages = 1, despite not being able to
With each failed allocation attempt incrementing the used counter, the subpool eventually reaches a point where its used counter equals its max counter. At that point, any future allocations that try to allocate hugeTLB pages from the subpool will fail, despite the subpool not having any of its hugeTLB pages consumed by any user.
Once this happens, there is no way to make the subpool usable again, since there is no way to decrement the used counter as no process is really consuming the hugeTLB pages.
The underflow issue that the original commit fixes still remains fixed as well.
Without this fix, used_hpages would keep on leaking if hugetlb_acct_memory() fails.