Nodemailer: Quadratic backtracking in SMTP AUTH mechanism detection regexes allows pre-auth remote denial of service

Summary

Nodemailer's SMTP client applies four regular expressions to a peer's EHLO response to detect the supported SASL mechanisms. Each pattern uses an alternation with two overlapping whitespace-matching sub-patterns, which produces quadratic (O(n^2)) backtracking whenever the mechanism name is absent from the line. A malicious or MITM SMTP server that answers EHLO with 250-AUTH followed by a long whitespace-only run and a non-matching token blocks the Node.js event loop for tens of seconds to hours during a pre-authentication handshake. No authentication to the target SMTP server is required, and any application that connects to an attacker-influenced host is affected.

Root Cause

Each of the four AUTH mechanism regexes in SMTPConnection._actionEHLO contains the sub-pattern (?:(\s+|=)[^\n]*\s+|\s+|=) before the mechanism literal. [^\n]* and the trailing \s+ both match whitespace and can consume the same characters, so when the mechanism literal is absent, the regex engine must try every partition of the whitespace run between [^\n]* and \s+ before failing. The four regexes run unconditionally on every EHLO response, with no length cap, no shape validation, and no early rejection before the engine enters the ambiguous alternation.

Component: nodemailer (npm), SMTP client
Function:  SMTPConnection._actionEHLO
Location:  src/smtp-connection/index.ts, lines 2162, 2167, 2172, 2177
Patterns:  /[ -]AUTH(?:(\s+|=)[^\n]*\s+|\s+|=)PLAIN/i
           /[ -]AUTH(?:(\s+|=)[^\n]*\s+|\s+|=)LOGIN/i
           /[ -]AUTH(?:(\s+|=)[^\n]*\s+|\s+|=)CRAM-MD5/i
           /[ -]AUTH(?:(\s+|=)[^\n]*\s+|\s+|=)XOAUTH2/i

Impact

Algorithmic-complexity denial of service against any Node.js process that opens an SMTP connection to an attacker-controlled or attacker-influenced host. Node.js is single threaded, so the process stalls entirely while the regex engine backtracks: no other requests are served, no timers fire, and the built-in socket timeout (10 minutes) cannot interrupt synchronous JavaScript. A 4,000-character whitespace run inside a single continuation line already blocks each regex for tens of seconds; the client bounds a response at 1 MB, which leaves headroom for blocking runs measured in hours across the four patterns. Two realistic attacker positions reach the sink pre-authentication: a hosted mailer that lets tenants configure their own SMTP server, and any network-adjacent attacker able to man-in-the-middle a cleartext SMTP connection (including before STARTTLS negotiation).

Proof of Concept

The PoC exercises the exact four regexes copied from src/smtp-connection/index.ts against a synthesized 250-AUTH<whitespace>NOMATCH line, so no live SMTP server is needed to reproduce the timing.

Runtime:      Node.js >= 20
Dependencies: none
Invocation:   node poc.js

'use strict';

const AUTH_REGEXES = [
    /[ -]AUTH(?:(\s+|=)[^\n]*\s+|\s+|=)PLAIN/i,
    /[ -]AUTH(?:(\s+|=)[^\n]*\s+|\s+|=)LOGIN/i,
    /[ -]AUTH(?:(\s+|=)[^\n]*\s+|\s+|=)CRAM-MD5/i,
    /[ -]AUTH(?:(\s+|=)[^\n]*\s+|\s+|=)XOAUTH2/i,
];

function bench(n) {
    const line = '250-AUTH' + ' '.repeat(n) + 'NOMATCH';
    const start = Date.now();
    for (const re of AUTH_REGEXES) re.test(line);
    return Date.now() - start;
}

for (const n of [500, 1000, 2000, 3000]) {
    console.log(`n=${n}  four-regex time=${bench(n)} ms`);
}

const control = '250-AUTH PLAIN LOGIN CRAM-MD5 XOAUTH2';
const t0 = Date.now();
for (const re of AUTH_REGEXES) re.(control);
.();

n=500   four-regex time=914 ms
n=1000  four-regex time=1348 ms
n=2000  four-regex time=10558 ms
n=3000  four-regex time=35251 ms
well-formed control  four-regex time=0 ms

Doubling the whitespace run multiplies the wall time by roughly four, which matches the expected O(n^2) growth. The well-formed control shows that a normal EHLO response is parsed in effectively zero time.

Attack Chain

1. Exposure: The victim application, or any dependent that uses nodemailer to send mail, opens a TCP or TLS connection to an SMTP host the attacker controls. This happens automatically for multi-tenant mailers whose tenants configure their own outbound SMTP server, and for any client whose plaintext SMTP session can be intercepted by a network-adjacent attacker before STARTTLS negotiation.
2. Control: The attacker fully controls the bytes returned in response to EHLO. In particular, the attacker chooses the length and content of every 250- continuation line up to the client's 1 MB response cap.
3. Path: Socket bytes are reassembled by _onData, decoded by _processResponse, and dispatched to _actionEHLO. On the first EHLO after the greeting (malicious server with no STARTTLS), on the second EHLO after a successful STARTTLS upgrade, or on the first EHLO when ignoreTLS is set, control reaches the AUTH detection block near the end of _actionEHLO.
4. Guard: The only bounds on this path are maxResponseSize (1 MB by default) and the 421 or non-2xx status-code check. Neither prevents O(n^2) backtracking: 4,000 whitespace characters already block each regex for tens of seconds, well below the 1 MB cap. No length, shape, or sanitization check runs before the regexes.
5. Primitive: Four RegExp.prototype.test calls execute against the crafted line, each entering catastrophic backtracking because [^\n]* and share the whitespace alphabet and the mechanism literal never matches.

Bypass Evidence

The only response-side protection in this code path is maxResponseSize, introduced by the fix for GHSA-4g23-2xm8-66gc (commit f2d82fa). That fix targeted quadratic reassembly of long multiline replies inside _onData and left the AUTH regex evaluation in _actionEHLO untouched. maxResponseSize does not prevent this issue: at 4,000 whitespace characters, roughly 0.4 percent of the default 1 MB cap, a single continuation line already blocks each of the four regexes for tens of seconds. No status-code or length gate rejects the payload before the regex engine enters the ambiguous alternation, and Node.js's single-threaded event loop cannot service the socket timeout while a synchronous test() call is running. Two other recent linear-time fixes in this project (the addressparser fallback for GHSA-v53p-9fqp-m79j and the DKIM header unfolding for GHSA-39m8-27wv-hr27) address entirely different modules and inputs and do not touch the AUTH mechanism regexes.

Affected Versions

• Ecosystem: npm
• Package: nodemailer
• Confirmed affected range: >= 3.0.0, <= 10.0.12
• Latest release checked: 10.0.12 (npm registry)
• Fix status: 10.0.13

The four AUTH mechanism regexes were introduced when the SMTP client was bundled into nodemailer at 3.0.0 (previously a separate smtp-connection package). The vulnerable alternation (?:(\s+|=)[^\n]*\s+|\s+|=) is present, unchanged, in every published release from 3.0.0 (lib/smtp-connection/index.js) through 10.0.12 (dist/esm/smtp-connection/index.js and dist/cjs/smtp-connection/index.js). A later refactor added a leading [ -] character class but did not alter the ambiguous inner alternation. No release, and no development branch inspected at the current HEAD (30e4cfd), contains a fix for this pattern.

Suggested Fix

Replace the four backtracking regexes with a non-ambiguous parse of the AUTH extension. The invariant the fix must enforce is that mechanism detection runs in time linear in the response length regardless of whitespace shape. A minimal approach: match the AUTH extension value