In the Linux kernel, the following vulnerability has been resolved:
btrfs: avoid potential out-of-bounds in btrfs_encode_fh()
The function btrfs_encode_fh() does not properly account for the three cases it handles.
Before writing to the file handle (fh), the function only returns to the user BTRFS_FID_SIZE_NON_CONNECTABLE (5 dwords, 20 bytes) or BTRFS_FID_SIZE_CONNECTABLE (8 dwords, 32 bytes).
However, when a parent exists and the root ID of the parent and the inode are different, the function writes BTRFS_FID_SIZE_CONNECTABLE_ROOT (10 dwords, 40 bytes).
If *max_len is not large enough, this write goes out of bounds because BTRFS_FID_SIZE_CONNECTABLE_ROOT is greater than BTRFS_FID_SIZE_CONNECTABLE originally returned.
This results in an 8-byte out-of-bounds write at fid->parent_root_objectid = parent_root_id.
A previous attempt to fix this issue was made but was lost.
https://lore.kernel.org/all/[email protected]/
Although this issue does not seem to be easily triggerable, it is a potential memory corruption bug that should be fixed. This patch resolves the issue by ensuring the function returns the appropriate size for all three cases and validates that *max_len is large enough before writing any data.