In the Linux kernel, the following vulnerability has been resolved:
mm: memcg: stop reclaim when a limit update is superseded
kernfs serializes file operations only per open file, so separate open files can update the same memory.high or memory.max file concurrently. Both handlers store the new limit before synchronous reclaim, but continue to use the writer's local target in the reclaim loop. If another writer raises or removes the limit, the first writer can continue reclaiming toward a stale target.
For memory.max, this can leave the writer looping indefinitely once reclaim retries are exhausted. The OOM path sees sufficient margin under the current limit and returns true without killing, while the writer still compares usage against its stale target and records another OOM event.
Check the current limit at the start of each reclaim iteration and stop if it no longer matches the writer's target.
Reproducer:
Populate a cgroup with anonymous memory and disable swapping. Lower memory.max from one open file, then restore it to "max" through another open file after the new limit becomes visible.
Without the patch, the first writer remains blocked and repeatedly increments the OOM event counter. With the patch, it returns normally.
This was not motivated by a reported production workload. We found it through automated randomized testing for our cgroup observability work and reduced it to the reproducer above.