Junrar: RarVM VM_JMP maxOpCount bypass causes infinite CPU loop during RAR4 extraction

Summary

Junrar's RAR virtual machine can be forced into a deterministic infinite loop while extracting a crafted RAR4 archive. The RarVM.ExecuteCode loop is intended to be bounded by maxOpCount, but jump instructions can bypass this limit when setIP() is called with a target instruction pointer greater than or equal to codeSize.

An unauthenticated attacker who can provide a RAR archive to an application using Junrar.extract() can cause the extraction thread to spin forever, consuming CPU and blocking the process. The issue is a denial of service vulnerability, not remote code execution.

Details

The root cause is in src/main/java/com/github/junrar/unpack/vm/RarVM.java.

setIP() returns early when the target IP is outside the prepared VM program:

private boolean setIP(int ip) {
    if ((ip) >= codeSize) {
        return (true);
    }

    if (--maxOpCount <= 0) {
        return (false);
    }

    IP = ip;
    return true;
}

When ip >= codeSize, this branch:

• does not update IP;
• does not decrement maxOpCount;
• returns true, so callers treat it as success.

The VM_JMP handler and the other jump-like handlers discard the return value of setIP() and immediately continue the main VM loop:

case VM_JMP:
    setIP(getValue(false, mem, op1));
    continue;

Because of the continue, execution also skips the loop-end accounting:

IP++;
--maxOpCount;

As a result, a VM program containing a single VM_JMP instruction can freeze both IP and maxOpCount:

1. IP points to the VM_JMP instruction.
2. The jump target resolves to codeSize or larger.
3. setIP() returns before modifying IP or decrementing maxOpCount.
4. The caller ignores the return value and continues.
5. The same VM_JMP instruction is executed again forever.

Impact

This is a CPU denial of service vulnerability. Any application that uses junrar to extract untrusted RAR archives can have an extraction worker thread permanently blocked by a small crafted archive. In services that process uploads or attachments automatically, repeated submissions can exhaust worker capacity and degrade or stop archive-processing functionality.

The RAR VM does not provide native code execution, reflection, or process execution primitives, and Java array bounds checks prevent escaping the VM memory array. The expected security impact is availability only.