---
id: "CVE-2026-126366"
page: "https://infrastack.ai/live/CVE-2026-126366"
kind: "vulnerability"
aliases: ["CVE-2026-126366", "GHSA-73v3-wf97-vpgj"]
title: "msgpack-ruby JRuby decoder makes uncapped header-count allocations; a 5-byte message forces multi-GiB OOM"
severity: "high"
cvss: 7.5
epss: null
exploited: false
status: "open"
category: "package"
products: [{"name": "msgpack", "fixed": ["1.8.7"], "affected": "<= 1.8.6", "ecosystem": "rubygems"}]
ecosystems: ["rubygems"]
tags: ["cwe-1325", "cwe-770", "java", "jruby", "logstash", "msgpack", "package", "rails", "rubygems"]
not_affected_if: ["cruby-only", "no-untrusted-msgpack-input"]
published_at: "2026-11-12T22:54:59+00:00"
first_seen_at: "2026-11-12T22:57:32+00:00"
updated_at: "2026-11-12T22:59:09+00:00"
sources: [{"source": "ghsa", "url": "https://github.com/advisories/GHSA-73v3-wf97-vpgj", "first_seen_at": "2026-11-12T22:57:32+00:00"}]
exploit_sources: []
---

# msgpack-ruby JRuby decoder makes uncapped header-count allocations; a 5-byte message forces multi-GiB OOM

Link for people: https://infrastack.ai/live/CVE-2026-126366 (this .md is for agents)

The Java (JRuby) decoder of the msgpack gem allocates memory based on element counts declared in the message header without any cap, so a 5-byte message can make the JVM attempt multi-gigabyte allocations. An attacker who can send msgpack data to a JRuby process can exhaust its heap and cause an out-of-memory failure, affecting availability.

_Summary, check and fix written by a model from the sources; labelled as such._

## Affected

- rubygems:msgpack affected <= 1.8.6 fixed 1.8.7

## How to check

- Run `jruby -S gem list msgpack` and check whether the java-platform msgpack version is `1.8.6` or earlier.
- Check whether your application feeds untrusted msgpack input to `MessagePack.unpack` or `MessagePack::Unpacker` under JRuby.

## Fix

- msgpack (rubygems, java platform): move to `1.8.7`, the fixed version in the product data.

## Timeline

- 2026-11-12T22:57:32+00:00 ghsa https://github.com/advisories/GHSA-73v3-wf97-vpgj

## Documents

- [advisory] Maintainer's advisory: api.github.com https://api.github.com/repos/msgpack/msgpack-ruby/security-advisories/GHSA-73v3-wf97-vpgj
- [advisory] Maintainer's advisory: github.com https://github.com/msgpack/msgpack-ruby/security/advisories/GHSA-73v3-wf97-vpgj
- [fix] Patch: github.com https://github.com/msgpack/msgpack-ruby/commit/ca0efc1256c687b428c46dd68ce000ff515d53f2
- [fix] Release: github.com https://github.com/msgpack/msgpack-ruby/releases/tag/v1.8.6
- [fix] Release: github.com https://github.com/msgpack/msgpack-ruby/releases/tag/v1.8.7

## From the sources

### Summary

The **JRuby** implementation of msgpack (`msgpack` gem, `java` platform — pure Java decoder under `ext/java/`) sizes containers directly from the element count written in the message header, with **no cap at all**: no equivalent of the C unpacker's `SHRT_MAX` clamp (since 1.7.3) and none of the buffered-bytes cap that commit c8d42d8 added for CRuby in 1.8.6. A single 5-byte message decides how much memory a JVM process tries to allocate:

* `array32` header declaring `0x7FFFFFFF` elements → `new IRubyObject[0x7FFFFFFF]` (~8 GiB of reference slots) attempted immediately → `java.lang.OutOfMemoryError: Requested array size exceeds VM limit`.
* Counts *below* the VM's max-array-size but above available heap (e.g. 5×10⁸ with a 512 MiB heap) produce a **real allocation attempt** — `OutOfMemoryError: Java heap space` — the JVM genuinely tries to take the memory before failing.
* `str32`/`bin32` headers declaring `0x7FFFFFFF` bytes do the same via `readBytes(size)` → `new byte[size]`.
* Below the failure threshold there is a **silent band**: one streaming unpacker fed only an `array32` header with count 2×10⁷ (5 wire bytes, zero elements) grows used heap **+78 MiB immediately**, no exception (`each` swallows the underflow), and the footprint stays elevated after `Unpacker#reset` + full GC. Linear in declared count (~4 B/element); nothing bounds it — not delivered bytes, not a per-container cap.

Only counts ≥ 2³¹ are refused (a clean `MessagePack::UnderflowError` from `getUnsignedInt()`), and only **maps** are safe (`consumeHash` grows per element instead of pre-sizing). Verified on the released gem `msgpack 1.8.6 java`, JRuby 9.4.12 / OpenJDK 25, 2026-10-30 — i.e. **unfixed in the current release**. Proposed fix mirrors c8d42d8: cap initial allocation by bytes actually buffered and grow as data arrives; read strings in bounded chunks instead of allocating the declared length up front.


### Details

### 1. The four dispatch arms, and which pre-allocate (line numbers at tag v1.8.6 — current release; `ext/java` has no commits after it)

| header | dispatch ([Decoder.java](https://github.com/msgpack/msgpack-ruby/blob/v1.8.6/ext/java/org/msgpack/jruby/Decoder.java)) | behaviour on the declared count | measured (released 1.8.6-java, Xmx512m) |
|---|---|---|---|
| `array32` `\xdd` | [:296](https://github.com/msgpack/msgpack-ruby/blob/v1.8.6/ext/java/org/msgpack/jruby/Decoder.java#L296) → [`consumeArray(int size)` :139-145](https://github.com/msgpack/msgpack-ruby/blob/v1.8.6/ext/java/org/msgpack/jruby/Decoder.java#L139-L145) | `new IRubyObject[size]` **immediately**, before reading any element | count `0x7FFFFFFF`: OOME "Requested array size exceeds VM limit"; count 5×10⁸: OOME "**Java heap space**" (real allocation attempt); count 2×10⁷: **+78 MiB, silent** |
| `str32` `\xdb` | [:294](https://github.com/msgpack/msgpack-ruby/blob/v1.8.6/ext/java/org/msgpack/jruby/Decoder.java#L294) → [`consumeString` :129](https://github.com/msgpack/msgpack-ruby/blob/v1.8.6/ext/java/org/msgpack/jruby/Decoder.java#L129) → [`readBytes(int size)` :186-190](https://github.com/msgpack/msgpack-ruby/blob/v1.8.6/ext/java/org/msgpack/jruby/Decoder.java#L186-L190) | `new byte[size]` **immediately** | len `0x7FFFFFFF`: OOME "array size exceeds VM limit" |
| `bin32` `\xc6` | [:269](https://github.com/msgpack/msgpack-ruby/blob/v1.8.6/ext/java/org/msgpack/jruby/Decoder.java#L269) → same `consumeString`/`readBytes` | same | len `0x7FFFFFFF`: OOME "array size exceeds VM limit" |
| `map32` `\xdf` | [:298](https://github.com/msgpack/msgpack-ruby/blob/v1.8.6/ext/java/org/msgpack/jruby/Decoder.java#L298) → [`consumeHash(int size)` :147+](https://github.com/msgpack/msgpack-ruby/blob/v1.8.6/ext/java/org/msgpack/jruby/Decoder.java#L147) | **no pre-allocation** — inserts per element, underflows on first missing byte | count `0x7FFFFFFF`: no exception (swallowed), no large allocation |

The only guard is [`getUnsignedInt()` :88-94](https://github.com/msgpack/msgpack-ruby/blob/v1.8.6/ext/java/org/msgpack/jruby/Decoder.java#L88-L94): counts with the top bit set (≥ 2³¹) raise `MessagePack::UnderflowError: Size too large (limited to 2**31 for the Java version)`. That bounds the *value* but leaves the entire positive band `(available heap, 2³¹)` unbounded — and that band is where real allocation attempts happen.

### 2. Contrast with the C implementation (same repo)

The CRuby unpacker has had a per-container cap since 1.7.3 (`INITIAL_BUFFER_CAPACITY_MAX = SHRT_MAX`) and, since [c8d42d8](https://github.com/msgpack/msgpack-ruby/commit/c8d42d8) in **1.8.6**, additionally caps initial size by bytes actually buffered ("an element needs at least one byte…"). Strings on C are bounded too (`read_raw_body_cont` allocates `min(remaining, buffered)`). The Java decoder has **neither generation of cap** — it is strictly behind the CRuby behaviour this repo already considers correct enough to ship.

### 3. Silent retention + mechanism control (measured 2026-10-30, released gem)

One `MessagePack::Unpacker` per process, fed only a header (`"\xdd" + [count].pack('N')`), zero elements; heap = `Runtime.total_memory - free_memory` after `GC.start` on both sides:

| declared count | used heap before → after | delta | exception? |
|---|---|---|---|
| 1,000,000 | 26.4 → 31.1 MiB | +4.7 | none (`each` swallows the underflow) |
| 5,000,000 | 53.3 → 74.0 MiB | +20.7 | none |
| 20,000,000 | 51.3 → 129.3 MiB | **+78.0** | none |

Mechanism control: after the feed, `Unpacker#reset` + three `GC.start` cycles left used heap at **+79.4 MiB above baseline** (2×10⁷ arm) — the elevation is *not* unpacker-held state; it is JVM heap already spent on header-declared allocation and not returned. Either way the process footprint grows per attack and stays grown; the peak cost is paid synchronously during parsing regardless of GC behaviour afterwards. (Honest scope note: this is unbounded **allocation**, not a memory leak.)

### 4. Why JVM makes availability impact worse than the CRuby analogue

In CRuby, allocation failure raises `NoMemoryError`/fails inside one worker process. In a JVM, an OOME raised on the parser thread can land anywhere in the host application's shared runtime — long-lived JRuby servers (Rails, Logstash-class pipelines, JDBC apps) share heap across all threads and connections; a single crafted message can destabilize or kill the whole JVM, not just the request. The silent band means no exception at all is even required to spend memory.

### 5. Not claimed / scope

* **Fluentd is NOT affected** (and this report deliberately does not involve it): Fluentd ships CRuby-only — CI pins `engine: cruby`, gemspec requires ruby >= 3.3 while JRuby 9.4 implements Ruby 3.1 semantics, and there are zero JRUBY guards in lib/. This is an upstream msgpack-ruby report only.
* Maps (`map32`) do not pre-allocate — measured; the class applies to `array32`/`str32`/`bin32`.
* Counts ≥ 2³¹ are cleanly refused — measured; no claim there.
* Older java-gem releases were not individually probed; `<= 1.8.6` means "every release up to the current one carries this code".


### Proof of concept

Verified on the **released gem** `msgpack 1.8.6 java`, JRuby 9.4.12.0, OpenJDK 25.0.4.1, Linux x86-64 (artifacts in `evidence_186_recheck/`). Always run with a small `-Xmx` so oversized requests fail fast instead of wrecking the box.

### Setup

```console
$ curl -fsSLO https://repo1.maven.org/maven2/org/jruby/jruby-dist/9.4.12.0/jruby-dist-9.4.12.0-bin.tar.gz
$ tar xzf jruby-dist-9.4.12.0-bin.tar.gz && export PATH=$PWD/jruby-9.4.12.0/bin:$PATH
$ jruby -S gem install msgpack --no-document      # resolves the java-platform gem: "msgpack (1.8.6 java)"
```

### 1. Header shapes (fresh process per arm, `-Xmx512m`)

```console
array32 count=0xFFFFFFFF       -> MessagePack::UnderflowError: Size too large (limited to 2**31 for the Java version)
array32 count=0x7FFFFFFF       -> Java::JavaLang::OutOfMemoryError: Requested array size exceeds VM limit
array32 cou

---

_Free to read, with fair use of about 60 pages a minute. For higher limits or API access: sales@infrastack.ai_