Pillow: Double free in ImagingScaleAffine on calloc failure via Image.transform

============================================================================
Double free in ImagingScaleAffine on calloc failure via Image.transform
Location: src/libImaging/Geometry.c:866  in ImagingScaleAffine()
Commit:   a707c6b21c313880d0771e54965d142dd42358b6
============================================================================

There is a double free in Pillow's Image.transform() that crashes the process when an oversized transform runs out of memory. Instead of only raising MemoryError, Pillow frees the output image's memory once while handling the error and then again when the Python object is cleaned up. Anyone who can choose the output size of Image.transform() can trigger it, but only on a worker with a memory limit (a common hardening for image workers) or a 32-bit process. The result is a crash of the worker, plus corruption of its heap that has not been shown to be controllable.

The bug is in ImagingScaleAffine() in src/libImaging/Geometry.c (line 866). When its calloc() for the per-column lookup table of 4 × width bytes fails, it calls ImagingDelete(imOut). When the call comes from Image.transform(), imOut is the Imaging struct owned by the live Python ImagingCore output object. _dealloc() in src/_imaging.c later runs ImagingDelete() on the freed struct again, writing its mutex and refcount in freed memory and freeing the pixel blocks a second time. The path is reached with Transform.EXTENT, or Transform.AFFINE with a pure scale/translate matrix, under the default NEAREST resampling. A very wide, 1-row output size makes the lookup table the allocation that fails while the image itself still fits. On 64-bit Linux with default overcommit, that requires an address-space limit such as ulimit -v, uwsgi --limit-as or systemd LimitAS=. resize(), thumbnail() and rotate() are not affected. The attached reproducer runs on a stock build with a 1 GB limit: transform((300000000, 1), EXTENT) raises and catches MemoryError, then the process dies with SIGSEGV in free() called from _dealloc(). A memory-checking build reports the same path as a heap use-after-free. The faulty line was inherited from PIL 1.1.7 in the original fork commit, so every Pillow release since 1.0 is affected. A patch is attached.

Details

ImagingScaleAffine() frees an image it does not own. On the one error path where its calloc() for the per-column lookup table fails, it calls ImagingDelete(imOut). When reached through Image.transform(), imOut is the Imaging struct held by a live Python ImagingCore object; that object's _dealloc() later calls ImagingDelete() on the already-freed struct and frees its pixel blocks a second time.

Shortest demonstration on the stock build (the address-space limit stands in for ulimit -v / uwsgi --limit-as / systemd LimitAS= on a worker):

import resource, gc
from PIL import Image
resource.setrlimit(resource.RLIMIT_AS, (1_000_000_000,) * 2)
src = Image.new("L", (8, 8))
try:
    src.transform((300_000_000, 1), Image.Transform.EXTENT, (0, 0, 8, 8))
except MemoryError:
    pass                      # expected, and the process should carry on
gc.collect()                  # instead: SIGSEGV in _dealloc -> ImagingDelete -> free()

The 300 MB output image allocates; calloc(300000000, 4) (1.2 GB) does not. MemoryError is raised as intended, but the interpreter then dies with SIGSEGV, gdb showing free() ← memory_return_block/ImagingDestroyArray (src/libImaging/Storage.c:396,408) ← ImagingDelete (Storage.c:271) ← _dealloc (src/_imaging.c:188). An AddressSanitizer build of the same sources (with max_allocation_size_mb used to make the calloc fail instead of an rlimit) reports heap-use-after-free in ImagingDelete src/libImaging/Storage.c:257 called from _dealloc src/_imaging.c:188, on a 128-byte region "freed by" ImagingDelete ← ImagingScaleAffine src/libImaging/Geometry.c:866 ← ImagingTransformAffine Geometry.c:1043 ← ← , "previously allocated by" ← … ← .

Code path

Image.transform() creates the output image as an ordinary Python-owned image and then transforms into it:

# src/PIL/Image.py:3008…3021
        im = new(self.mode, size, fillcolor)
        …
            im.__transformer(
                (0, 0, *size), self, method, data, resample, fillcolor is None
            )

__transformer() rewrites Transform.EXTENT into an axis-aligned affine matrix (src/PIL/Image.py:3040-3046: data = (xs, 0, x0, 0, ys, y0), method = Transform.AFFINE) and calls self.im.transform(box, image.im, method, data, resample, fill) (Image.py:3117). resample defaults to Resampling.NEAREST (Image.py:2936).

In C, _transform() passes its own image as the destination:

/* src/_imaging.c:2153 */
    imOut = ImagingTransform(
        self->image, imagep->image, method, x0, y0, x1, y1, a, filter, fill
    );

ImagingTransformAffine() takes the scaling fast path because the filter is NEAREST and the matrix has no shear:

/* src/libImaging/Geometry.c:1035-1043 */
    if (filterid || imIn->type == IMAGING_TYPE_I16) {
        return ImagingGenericTransform(…);
    }
    if (a[1] == 0 && a[3] == 0) {
        /* Scaling */
        return ImagingScaleAffine(imOut, imIn, x0, y0, x1, y1, a, fill);
    }

and ImagingScaleAffine() allocates a table of imOut->xsize ints:

/* src/libImaging/Geometry.c:863-868 */
    /* malloc check ok, uses calloc for overflow */
    xintab = (int *)calloc(imOut->xsize, sizeof(int));
    if (!xintab) {
        ImagingDelete(imOut);                         /* <-- imOut belongs to the caller */
        return (Imaging)ImagingError_MemoryError();
    }

ImagingDelete() (src/libImaging/Storage.c:250-279) drops the refcount to 0, returns the pixel blocks to the arena / free()s them via im->destroy, and free(im). self->image in the ImagingCore object still points at the freed struct. _transform() returns NULL, Image.transform() propagates MemoryError, the half-built output Image loses its last reference, and _dealloc() (src/_imaging.c:188) calls ImagingDelete(imagep->image) again: it locks im->mutex and decrements im->refcount inside freed memory, then walks im->blocks (already freed, typically reused by then) and frees every entry a second time. In the observed stock-build runs the freed array had been overwritten, so receives a mangled pointer and the process segfaults; with other heap states it is a plain double free of a 300 MB block.

Every other error return in Geometry.c (ImagingError_ModeError, ImagingError_Mismatch, "bad filter number", "bad transform method") leaves imOut alone; line 866 is the only one that deletes it. The delete is only correct for the other caller, _resize() (src/_imaging.c:2026-2030), where imOut is a fresh ImagingNewDirty() temporary that nothing else references. The delete predates Pillow's git history: it came in with the PIL 1.1.7 import, 9a640e315715 ("Forking PIL", 2010-07-30), so every release since Pillow 1.0 is affected.

The output width is what decides whether the path is hit: the output image costs width × height × pixelsize bytes, the table an additional 4 × width. A very wide, 1-row output (300 000 000 × 1 "L": 300 MB image, 1.2 GB table) makes the table the allocation that fails. ImagingNewPrologueSubtype() caps xsize at INT_MAX/4 - 1, so the table is at most ~2 GB.

Impact

Applications that let a user choose the output size of Image.transform() — Transform.EXTENT (crop-and-scale), Transform.AFFINE with a pure scale/translate matrix, or the ImageTransform.ExtentTransform/AffineTransform helpers, all with the default NEAREST resampling — can have an allocation failure turned i