Unbounded memory growth in the NestJS TCP microservice transport

The same code is present in the 10.x line and earlier. Those lines are end-of-life and will not receive a patch; upgrade to a supported major.

Only @nestjs/microservices is affected. No other package needs updating for this issue.

Details

Two independent paths let one peer grow the heap without bound.

Partial packets are buffered with nothing to reap them

The TCP transport frames messages as <length>#<payload>. A peer may declare a length, send part of the payload, and then stop. JsonSocket keeps the partial payload buffered while it waits for the rest, and nothing ever reclaimed it: there was no socket timeout, no cap on the number of connections, and no tracking of accepted sockets.

maxBufferSize did not close this. It caps a single connection, defaulting to 128M characters, and is enforced per connection, so the effective ceiling was the number of connections an attacker chose to open.

Ten connections that each send 20MB and then go silent:

The memory was held for as long as the connections stayed open. Closing them released it, so a peer could hold it indefinitely at negligible cost to itself.

Response backpressure was ignored

JsonSocket#handleSend discarded the return value of socket.write. A peer that issues requests and never reads the responses therefore made the process queue every response in memory, with no ceiling and no signal to stop producing more.

Thirty requests from a client whose stream is paused, against a handler returning an 8MB payload, grew rss from 152.4MB to 419.1MB. The amplification here is roughly 1:1, because the response size is set by the application rather than by the attacker; the defect is that the server buffers all of it rather than applying backpressure or giving up.

Impact

An unauthenticated peer that can reach the transport's port can drive the process to an out-of-memory kill. Under a container memory limit this is fast and repeatable, and it restarts the pod rather than merely degrading it.

There is no confidentiality or integrity impact. Nothing is read, written, or executed; the failure mode is availability only.

Applications are not affected if the TCP transport is not used, or if the transport's port is reachable only by trusted peers.

Patches

Upgrade @nestjs/microservices to 12.0.3 or 11.2.5.

Three changes landed together:

• A stall timer drops a peer that stops sending mid-packet. It is armed only while a packet is partially received and is refreshed on every read, so a slow but progressing transfer is never interrupted and a peer sitting idle between packets is left alone. Configurable as incompleteMessageTimeout, default 30000ms.
• Reading is suspended while a peer's outgoing buffer is backed up and resumes on drain, so a peer that stops reading cannot keep issuing requests.
• Responses queued for a peer that reads nothing are capped. Suspending reads is not sufficient on its own, because message handlers are asynchronous: a burst of pipelined requests is dispatched before the first response is written, and those responses still queue. Configurable as maxSendBufferSize, default 128MB.

Both limits are configurable on the server and on the client, and either can be disabled by setting it to 0.

Behaviour change

Both limits are enabled by default, following the precedent set by maxBufferSize. A peer that stops sending mid-packet for 30 seconds, or that lets more than 128MB of responses queue unread on a single connection, is now disconnected where previously it was not.

The send cap can also drop a peer that is genuinely reading, if responses are produced faster than the socket drains for long enough. The default is set high for that reason. Applications that stream very large responses to deliberately slow consumers should raise maxSendBufferSize or set it to 0.

Workarounds

For anyone who cannot upgrade immediately:

• Restrict who can reach the port. The TCP transport is designed for trusted service-to-service traffic. A network policy, security group, or L4 firewall that limits the port to known peers removes the exposure entirely, and is worth doing regardless of version.
• Lower maxBufferSize. This reduces what a single connection can pin on the receive side. It does not bound the total, because the limit is per connection and the number of connections is not capped, and it does nothing for the response-queueing path.
• Supply a custom socketClass. An application-provided socket class can implement its own idle timeout and backpressure handling. This is the only pre-patch mitigation that addresses the sending side.

Setting a process memory limit does not mitigate this. It converts the failure from an out-of-memory kill of the host into an out-of-memory kill of the process.

Credit

Reported by Salman Aljardan, who provided a detailed write-up and self-contained proofs of concept for each issue, and coordinated disclosure.

References

• Fix, 12.x, receive and send side bounds: https://github.com/nestjs/nest/pull/17746
• Fix, 12.x, connection teardown on shutdown: https://github.com/nestjs/nest/pull/17745
• Backport, 11.x: https://github.com/nestjs/nest/pull/17749
• Release 12.0.3: https://github.com/nestjs/nest/releases/tag/v12.0.3