ASGI worker: nr_conns never decremented on server-initiated close — every graceful stop waits out the full graceful_timeout

[eduramirezh]

Environment

• gunicorn 26.0.0 (the code path is unchanged in current master)
• CPython 3.12, --worker-class asgi, plain HTTP/1.x
• gunicorn_h1c not installed (pure-Python callback parser path)

Symptom

Every worker stop, both SIGTERM and the --max-requests autorestart, blocks for the full graceful_timeout and then logs Forcing close of N connections, where N keeps growing with worker age. It happens even when the worker is completely idle.

Production example (3 workers, --max-requests 1000 --max-requests-jitter 500, behind an nginx that proxies HTTP/1.0 close-per-request), captured at a zero-traffic hour:

06:07:18 [132970] [INFO] Worker shutting down...
06:07:18 [132970] [INFO] Waiting for 1426 connections to finish...
06:07:28 [146478] [INFO] Waiting for 347 connections to finish...
06:07:28 [134827] [INFO] Waiting for 1350 connections to finish...
06:09:13 [132970] [WARNING] Forcing close of 1426 connections     <- 115s later, same number
06:09:13 [132970] [INFO] Worker exiting (pid: 132970)

The counts match each worker's requests served since boot (1426 and 1350 are inside the 1000 to 1500 max-requests band; the worker logging 1426 was autorestarting for having reached it). They are not real connections: the count never moves during the whole graceful window.

Cause

In gunicorn/asgi/protocol.py:

• connection_made() does self.worker.nr_conns += 1.
• A server-initiated close goes through _close_transport(), which calls transport.close() and sets self._closed = True. It does not decrement.
• When asyncio then delivers connection_lost(), the idempotence guard returns before the decrement:

def connection_lost(self, exc):
    # Guard against multiple calls (idempotent)
    if self._closed:
        return

    self._closed = True
    self.worker.nr_conns -= 1

So every connection the server closes first leaks one count permanently. That covers any Connection: close response (for example HTTP/1.0 reverse proxies), every keepalive timeout (_keepalive_timeout() calls _close_transport()), and error aborts. Client-initiated closes are balanced because connection_lost() runs first, which is presumably why client-driven test suites do not catch it.

ASGIWorker._shutdown() (gunicorn/workers/gasgi.py) then waits on that counter:

if self.nr_conns > 0 and not self._quick_shutdown:
    graceful_timeout = self.cfg.graceful_timeout
    self.log.info("Waiting for %d connections to finish...", self.nr_conns)
    deadline = self.loop.time() + graceful_timeout
    while self.nr_conns > 0 and self.loop.time() < deadline:
        ...

Since the counter can never reach zero, every graceful stop runs to the deadline. Consequences: deploys that SIGTERM workers always take the full graceful_timeout, and each --max-requests recycle keeps a worker out of rotation (not accepting) for that whole window, which can starve the listen socket when several workers recycle close together.

Minimal repro (protocol level, no sockets)

import asyncio
import logging
import types

from gunicorn.asgi.protocol import ASGIProtocol
from gunicorn.config import Config


class FakeTransport:
    def get_extra_info(self, name, default=None):
        return default  # no ssl_object: plain HTTP/1.x path

    def set_write_buffer_limits(self, high=None, low=None):
        pass

    def can_write_eof(self):
        return False

    def close(self):
        pass

    def is_closing(self):
        return True


def make_worker():
    return types.SimpleNamespace(
        cfg=Config(),
        loop=asyncio.get_running_loop(),
        nr_conns=0,
        alive=True,
        log=logging.getLogger("repro"),
        asgi=None,
    )


async def main():
    # Server closes first (Connection: close response, keepalive timeout, ...)
    worker = make_worker()
    proto = ASGIProtocol(worker)
    proto.connection_made(FakeTransport())
    proto._close_transport()      # sets _closed = True
    proto.connection_lost(None)   # guard returns early: no decrement
    print("server-initiated close, nr_conns =", worker.nr_conns)  # 1  (leak)

    # Client closes first: balanced
    worker = make_worker()
    proto = ASGIProtocol(worker)
    proto.connection_made(FakeTransport())
    proto.connection_lost(None)
    proto._close_transport()
    print("client-initiated close, nr_conns =", worker.nr_conns)  # 0


asyncio.run(main())

Output on 26.0.0:

server-initiated close, nr_conns = 1
client-initiated close, nr_conns = 0

End-to-end repro

# app.py
async def app(scope, receive, send):
    if scope["type"] != "http":
        return
    await send({"type": "http.response.start", "status": 200, "headers": []})
    await send({"type": "http.response.body", "body": b"ok"})

$ gunicorn app:app --worker-class asgi --workers 1 --bind 127.0.0.1:8123 --graceful-timeout 115 &
$ for i in 1 2 3; do curl -s --http1.0 http://127.0.0.1:8123/ >/dev/null; done
$ kill -TERM %1

The worker logs Waiting for 3 connections to finish... with no client connected and exits only when the 115s deadline expires. With the decrement fixed (we currently carry the wrapper below in our gunicorn.conf.py), the same sequence exits in about 2 seconds.

Workaround / suggested fix

Perform the decrement wherever _closed flips, so each connection is decremented exactly once whichever side closes first. As a config-file workaround:

from gunicorn.asgi.protocol import ASGIProtocol

_orig = ASGIProtocol._close_transport

def _close_transport(self):
    was_closed = self._closed
    _orig(self)
    if self._closed and not was_closed:
        self.worker.nr_conns -= 1

ASGIProtocol._close_transport = _close_transport

A proper fix could decrement in _close_transport() itself, or centralize the close accounting in a single helper used by both paths. Similar accounting bug fixed long ago in the gthread worker: #1387.

Happy to open a PR.

[DresdenGman]

I can open a PR for this — the fix is clear from the analysis: decrement nr_conns in _close_transport() when the connection wasn't already closed, so the count stays balanced regardless of who initiates the close.